Ecg interpretation system

ABSTRACT

An electrocardiogram (ECG) interpretation system is operable to receive a captured image of an ECG printout. A waveform detection function is performed on the captured image to determine a plurality of locations of a plurality of ECG waveforms in the captured image. A plurality of waveform images are generated by partitioning the captured image based on the plurality of locations, where each of the plurality of waveform images includes one of the plurality of ECG waveforms. A plurality of pseudo-raw ECG signal data is generated by performing a signal reconstruction function on each of the plurality of waveform images, where each of the plurality of pseudo-raw ECG signal data corresponds to one of the plurality of waveform images. Diagnosis data is generated by performing a diagnosing function on the plurality of pseudo-raw ECG signal data. The diagnosis data is transmitted to a client device for display via a display device.

CROSS REFERENCE TO RELATED APPLICATIONS

The present U.S. Utility patent application claims priority pursuant to35 U.S.C. § 119(e) to U.S. Provisional Application No. 62/770,334,entitled “LESION TRACKING SYSTEM”, filed Nov. 21, 2018, which is herebyincorporated herein by reference in its entirety and made part of thepresent U.S. Utility patent application for all purposes.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not applicable.

INCORPORATION-BY-REFERENCE OF MATERIAL SUBMITTED ON A COMPACT DISC

Not applicable.

BACKGROUND Technical Field

This invention relates generally to medical imaging devices andknowledge-based systems used in conjunction with client/server networkarchitectures.

Description of Related Art

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWING(S)

FIG. 1 is a schematic block diagram of an embodiment of a medical scanprocessing system;

FIG. 2A is a schematic block diagram of a client device in accordancewith various embodiments;

FIG. 2B is a schematic block diagram of one or more subsystems inaccordance with various embodiments;

FIG. 3 is a schematic block diagram of a database storage system inaccordance with various embodiments;

FIG. 4A is schematic block diagram of a medical scan entry in accordancewith various embodiments;

FIG. 4B is a schematic block diagram of abnormality data in accordancewith various embodiments;

FIG. 5A is a schematic block diagram of a user profile entry inaccordance with various embodiments;

FIG. 5B is a schematic block diagram of a medical scan analysis functionentry in accordance with various embodiments;

FIGS. 6A-6B are schematic block diagram of a medical scan diagnosingsystem in accordance with various embodiments;

FIG. 7A is a flowchart representation of an inference step in accordancewith various embodiments;

FIG. 7B is a flowchart representation of a detection step in accordancewith various embodiments;

FIGS. 8A-8F are schematic block diagrams of a medical picture archiveintegration system in accordance with various embodiments;

FIG. 9 is a flowchart representation of a method for execution by amedical picture archive integration system in accordance with variousembodiments;

FIG. 10A is a schematic block diagram of a de-identification system inaccordance with various embodiments;

FIG. 10B is an illustration of an example of anonymizing patientidentifiers in image data of a medical scan in accordance with variousembodiments;

FIG. 11 presents a flowchart illustrating a method for execution by ade-identification system in accordance with various embodiments;

FIG. 12A is a schematic block diagram of a ECG interpretation system inaccordance with various embodiments;

FIG. 12B is a schematic block diagram of an ECG entry in accordance withvarious embodiments;

FIGS. 12C-12E present flowchart representations of performing steps of aECG processing function in accordance with various embodiments;

FIG. 12F is a schematic block diagram of an ECG interpretation system inaccordance with various embodiments;

FIGS. 13A-13E are illustrations of an example user interface displayedvia a display device in accordance with various embodiments;

FIG. 14 presents a flowchart illustrating a method for execution by anECG interpretation system in accordance with various embodiments;

FIG. 15A is a schematic block diagram of a captured image processingsystem in accordance with various embodiments;

FIGS. 15B-15C are illustrations of an example user interface displayedvia a display device in accordance with various embodiments;

FIG. 16A is a schematic block diagram of a lesion tracking system inaccordance with various embodiments;

FIG. 16B is an illustration of an example of a lesion diametermeasurement in accordance with various embodiments;

FIG. 16C is a flowchart illustration of performing a lesion volumemeasurement function in accordance with various embodiments; and

FIG. 16D is an illustration of an example interface displayed by adisplay device in accordance with various embodiments.

DETAILED DESCRIPTION

The present U.S. Utility patent application is related to U.S. Utilityapplication Ser. No. 15/627,644, entitled “MEDICAL SCAN ASSISTED REVIEWSYSTEM”, filed 20 Jun. 2017, which claims priority pursuant to 35 U.S.C.§ 119(e) to U.S. Provisional Application No. 62/511,150, entitled“MEDICAL SCAN ASSISTED REVIEW SYSTEM AND METHODS”, filed 25 May 2017,both of which are hereby incorporated herein by reference in theirentirety and made part of the present U.S. Utility patent applicationfor all purposes.

FIG. 1 presents a medical scan processing system 100, which can includeone or more medical scan subsystems 101 that communicate bidirectionallywith one or more client devices 120 via a wired and/or wireless network150. The medical scan subsystems 101 can include a medical scan assistedreview system 102, medical scan report labeling system 104, a medicalscan annotator system 106, a medical scan diagnosing system 108, amedical scan interface feature evaluator system 110, a medical scanimage analysis system 112, a medical scan natural language analysissystem 114, and/or a medical scan comparison system 116. Some or all ofthe subsystems 101 can utilize the same processing devices, memorydevices, and/or network interfaces, for example, running on a same setof shared servers connected to network 150. Alternatively or inaddition, some or all of the subsystems 101 be assigned their ownprocessing devices, memory devices, and/or network interfaces, forexample, running separately on different sets of servers connected tonetwork 150. Some or all of the subsystems 101 can interact directlywith each other, for example, where one subsystem's output istransmitted directly as input to another subsystem via network 150.Network 150 can include one or more wireless and/or wired communicationsystems; one or more non-public intranet systems and/or public internetsystems; and/or one or more local area networks (LAN) and/or wide areanetworks (WAN).

The medical scan processing system 100 can further include a databasestorage system 140, which can include one or more servers, one or morememory devices of one or more subsystems 101, and/or one or more othermemory devices connected to network 150. The database storage system 140can store one or more shared databases and/or one or more files storedon one or more memory devices that include database entries as describedherein. The shared databases and/or files can each be utilized by someor all of the subsystems of the medical scan processing system, allowingsome or all of the subsystems and/or client devices to retrieve, edit,add, or delete entries to the one or more databases and/or files.

The one or more client devices 120 can each be associated with one ormore users of one or more subsystems of the medical scan processingsystem. Some or all of the client devices can be associated withhospitals or other medical institutions and/or associated with medicalprofessionals, employees, or other individual users for example, locatedat one or more of the medical institutions. Some of the client devices120 can correspond to one or more administrators of one or moresubsystems of the medical scan processing system, allowingadministrators to manage, supervise, or override functions of one ormore subsystems for which they are responsible.

Some or all of the subsystems 101 of the medical scan processing system100 can include a server that presents a website for operation via abrowser of client devices 120. Alternatively or in addition, each clientdevice can store application data corresponding to some or allsubsystems, for example, a subset of the subsystems that are relevant tothe user in a memory of the client device, and a processor of the clientdevice can display the interactive interface based on instructions inthe interface data stored in memory. For example, the website presentedby a subsystem can operate via the application. Some or all of thewebsites presented can correspond to multiple subsystems, for example,where the multiple subsystems share the server presenting the website.Furthermore, the network 150 can be configured for secure and/orauthenticated communications between the medical scan subsystems 101,the client devices 120 and the database storage system 140 to protectthe data stored in the database storage system and the data communicatedbetween the medical scan subsystems 101, the client devices 120 and thedatabase storage system 140 from unauthorized access.

The medical scan assisted review system 102 can be used to aid medicalprofessionals or other users in diagnosing, triaging, classifying,ranking, and/or otherwise reviewing medical scans by presenting amedical scan for review by a user by transmitting medical scan data of aselected medical scan and/or interface feature data of selectedinterface features of to a client device 120 corresponding to a user ofthe medical scan assisted review system for display via a display deviceof the client device. The medical scan assisted review system 102 cangenerate scan review data for a medical scan based on user input to theinteractive interface displayed by the display device in response toprompts to provide the scan review data, for example, where the promptscorrespond to one or more interface features.

The medical scan assisted review system 102 can be operable to receive,via a network, a medical scan for review. Abnormality annotation datacan be generated by identifying one or more of abnormalities in themedical scan by utilizing a computer vision model that is trained on aplurality of training medical scans. The abnormality annotation data caninclude location data and classification data for each of the pluralityof abnormalities and/or data that facilitates the visualization of theabnormalities in the scan image data. Report data including textdescribing each of the plurality of abnormalities is generated based onthe abnormality data. The visualization and the report data, which cancollectively be displayed annotation data, can be transmitted to aclient device. A display device associated with the client device candisplay the visualization in conjunction with the medical scan via aninteractive interface, and the display device can further display thereport data via the interactive interface.

In various embodiments, longitudinal data, such as one or moreadditional scans of longitudinal data 433 of the medical scan or ofsimilar scans, can be displayed in conjunction with the medical scanautomatically, or in response to the user electing to view longitudinaldata via user input. For example, the medical scan assisted reviewsystem can retrieve a previous scan or a future scan for the patientfrom a patient database or from the medical scan database automaticallyor in response to the user electing to view past patient data. One ormore previous scans can be displayed in one or more correspondingwindows adjacent to the current medical scan. For example, the user canselect a past scan from the longitudinal data for display. Alternativelyor in addition, the user can elect longitudinal parameters such asamount of time elapsed, scan type, electing to select the most recentand/or least recent scan, electing to select a future scan, electing toselect a scan at a date closest to the scan, or other criteria, and themedical scan assisted review system can automatically select a previousscan that compares most favorably to the longitudinal parameters. Theselected additional scan can be displayed in an adjacent windowalongside the current medical scan. In some embodiments, multipleadditional scans will be selected and can be displayed in multipleadjacent windows.

In various embodiments, a first window displaying an image slice 412 ofthe medical scan and an adjacent second window displaying an image sliceof a selected additional scan will display image slices 412 determinedto correspond with the currently displayed slice 412 of the medicalscan. As described with respect to selecting a slice of a selectedsimilar medical scan for display, this can be achieved based onselecting the image slice with a matching slice number, based onautomatically determining the image slice that most closely matches theanatomical region corresponding to the currently displayed slice of thecurrent scan, and/or based on determining the slice in the previous scanwith the most similar view of the abnormality as the currently displayedslice. The user can use a single scroll bar or other single user inputindication to jump to a different image slice, and the multiple windowscan simultaneously display the same numbered image slice, or can scrollor jump by the same number of slices if different slice numbers areinitially displayed. In some embodiments, three or more adjacent windowscorresponding to the medical scan and two or more additional scans aredisplayed, and can all be controlled with the single scroll bar in asimilar fashion.

The medical scan assisted review system 102 can automatically detectprevious states of the identified abnormalities based on the abnormalitydata, such as the abnormality location data. The detected previousstates of the identified abnormality can be circled, highlighted, orotherwise indicated in their corresponding window. The medical scanassisted review system 102 can retrieve classification data for theprevious state of the abnormality by retrieving abnormality annotationdata 442 of the similar abnormality mapped to the previous scan from themedical scan database 342. This data may not be assigned to the previousscan, and the medical scan assisted review system can automaticallydetermine classification or other diagnosis data for the previousmedical scan by utilizing the medical scan image analysis system asdiscussed. Alternatively or in addition, some or all of the abnormalityclassification data 445 or other diagnosis data 440 for the previousscan can be assigned values determined based on the abnormalityclassification data or other diagnosis data determined for the currentscan. Such abnormality classification data 445 or other diagnosis data440 determined for the previous scan can be mapped to the previous scan,and or mapped to the longitudinal data 433, in the database and/ortransmitted to a responsible entity via the network.

The medical assisted review system can automatically generate statechange data such as a change in size, volume, malignancy, or otherchanges to various classifiers of the abnormality. This can be achievedby automatically comparing image data of one or more previous scans andthe current scan and/or by comparing abnormality data of the previousscan to abnormality data of the current scan. In some embodiments, suchmetrics can be calculated by utilizing the medical scan similarityanalysis function, for example, where the output of the medical scansimilarity analysis function such as the similarity score indicatesdistance, error, or other measured discrepancy in one or moreabnormality classifier categories 444 and/or abnormality patterncategories 446. This calculated distance, error, or other measureddiscrepancy in each category can be used to quantify state change data,indicate a new classifier in one or more categories, to determine if acertain category has become more or less severe, or otherwise determinehow the abnormality has changed over time. In various embodiments, thisdata can be displayed in one window, for example, where an increase inabnormality size is indicated by overlaying or highlighting an outlineof the current abnormality over the corresponding image slice of theprevious abnormality, or vice versa. In various embodiments whereseveral past scans are available, such state change data can bedetermined over time, and statistical data showing growth rate changesover time or malignancy changes over time can be generated, for example,indicating if a growth rate is lessening or worsening over time. Imageslices corresponding to multiple past scans can be displayed insequence, for example, where a first scroll bar allows a user to scrollbetween image slice numbers, and a second scroll bar allows a user toscroll between the same image slice over time. In various embodimentsthe abnormality data, heat map data, or other interface features will bedisplayed in conjunction with the image slices of the past image data.

The medical scan report labeling system 104 can be used to automaticallyassign medical codes to medical scans based on user identified keywords,phrases, or other relevant medical condition terms of natural text datain a medical scan report of the medical scan, identified by users of themedical scan report labeling system 104. The medical scan reportlabeling system 104 can be operable to transmit a medical report thatincludes natural language text to a first client device for display.Identified medical condition term data can be received from the firstclient device in response. An alias mapping pair in a medical labelalias database can be identified by determining that a medical conditionterm of the alias mapping pair compares favorably to the identifiedmedical condition term data. A medical code that corresponds to thealias mapping pair and a medical scan that corresponds to the medicalreport can be transmitted to a second client device of an expert userfor display, and accuracy data can be received from the second clientdevice in response. The medical code is mapped to the first medical scanin a medical scan database when the accuracy data indicates that themedical code compares favorably to the medical scan.

The medical scan annotator system 106 can be used to gather annotationsof medical scans based on review of the medical scan image data by usersof the system such as radiologists or other medical professionals.Medical scans that require annotation, for example, that have beentriaged from a hospital or other triaging entity, can be sent tomultiple users selected by the medical scan annotator system 106, andthe annotations received from the multiple medical professionals can beprocessed automatically by a processing system of the medical scanannotator system, allowing the medical scan annotator system toautomatically determine a consensus annotation of each medical scan.Furthermore, the users can be automatically scored by the medical scanannotator system based on how closely their annotation matches to theconsensus annotation or some other truth annotation, for example,corresponding to annotations of the medical scan assigned a truth flag.Users can be assigned automatically to annotate subsequent incomingmedical scans based on their overall scores and/or based on categorizedscores that correspond to an identified category of the incoming medicalscan.

The medical scan annotator system 106 can be operable to select amedical scan for transmission via a network to a first client device anda second client device for display via an interactive interface, andannotation data can be received from the first client device and thesecond client device in response. Annotation similarity data can begenerated by comparing the first annotation data to the secondannotation data, and consensus annotation data can be generated based onthe first annotation data and the second annotation data in response tothe annotation similarity data indicating that the difference betweenthe first annotation data and the second annotation data comparesfavorably to an annotation discrepancy threshold. The consensusannotation data can be mapped to the medical scan in a medical scandatabase.

A medical scan diagnosing system 108 can be used by hospitals, medicalprofessionals, or other medical entities to automatically produceinference data for given medical scans by utilizing computer visiontechniques and/or natural language processing techniques. Thisautomatically generated inference data can be used to generate and/orupdate diagnosis data or other corresponding data of correspondingmedical scan entries in a medical scan database. The medical scandiagnosing system can utilize a medical scan database, user database,and/or a medical scan analysis function database by communicating withthe database storage system 140 via the network 150, and/or can utilizeanother medical scan database, user database, and/or function databasestored in local memory.

The medical scan diagnosing system 108 can be operable to receive amedical scan. Diagnosis data of the medical scan can be generated byperforming a medical scan inference function on the medical scan. Thefirst medical scan can be transmitted to a first client deviceassociated with a user of the medical scan diagnosing system in responseto the diagnosis data indicating that the medical scan corresponds to anon-normal diagnosis. The medical scan can be displayed to the user viaan interactive interface displayed by a display device corresponding tothe first client device. Review data can be received from the firstclient device, where the review data is generated by the first clientdevice in response to a prompt via the interactive interface. Updateddiagnosis data can be generated based on the review data. The updateddiagnosis data can be transmitted to a second client device associatedwith a requesting entity.

A medical scan interface feature evaluating system 110 can be usedevaluate proposed interface features or currently used interfacefeatures of an interactive interface to present medical scans for reviewby medical professionals or other users of one or more subsystems 101.The medical scan interface feature evaluator system 110 can be operableto generate an ordered image-to-prompt mapping by selecting a set ofuser interface features to be displayed with each of an ordered set ofmedical scans. The set of medical scans and the ordered image-to-promptmapping can be transmitted to a set of client devices. A set ofresponses can be generated by each client device in response tosequentially displaying each of the set of medical scans in conjunctionwith a mapped user interface feature indicated in the orderedimage-to-prompt mapping via a user interface. Response score data can begenerated by comparing each response to truth annotation data of thecorresponding medical scan. Interface feature score data correspondingto each user interface feature can be generated based on aggregating theresponse score data, and is used to generate a ranking of the set ofuser interface features.

A medical scan image analysis system 112 can be used to generate and/orperform one or more medical scan image analysis functions by utilizing acomputer vision-based learning algorithm 1350 on a training set ofmedical scans with known annotation data, diagnosis data, labelingand/or medical code data, report data, patient history data, patientrisk factor data, and/or other metadata associated with medical scans.These medical scan image analysis functions can be used to generateinference data for new medical scans that are triaged or otherwiserequire inferred annotation data, diagnosis data, labeling and/ormedical code data, and/or report data. For example, some medical scanimage analysis functions can correspond to medical scan inferencefunctions of the medical scan diagnosing system or other medical scananalysis functions of a medical scan analysis function database. Themedical scan image analysis functions can be used to determine whetheror not a medical scan is normal, to detect the location of anabnormality in one or more slices of a medical scan, and/or tocharacterize a detected abnormality. The medical scan image analysissystem can be used to generate and/or perform computer vision basedmedical scan image analysis functions utilized by other subsystems ofthe medical scan processing system as described herein, aiding medicalprofessionals to diagnose patients and/or to generate further data andmodels to characterize medical scans. The medical scan image analysissystem can include a processing system that includes a processor and amemory that stores executable instructions that, when executed by theprocessing system, facilitate performance of operations.

The medical scan image analysis system 112 can be operable to receive aplurality of medical scans that represent a three-dimensional anatomicalregion and include a plurality of cross-sectional image slices. Aplurality of three-dimensional subregions corresponding to each of theplurality of medical scans can be generated by selecting a proper subsetof the plurality of cross-sectional image slices from each medical scan,and by further selecting a two-dimensional subregion from each propersubset of cross-sectional image slices. A learning algorithm can beperformed on the plurality of three-dimensional subregions to generate aneural network. Inference data corresponding to a new medical scanreceived via the network can be generated by performing an inferencealgorithm on the new medical scan by utilizing the neural network. Aninferred abnormality can be identified in the new medical scan based onthe inference data.

The medical scan natural language analysis system 114 can determine atraining set of medical scans with medical codes determined to be truthdata. Corresponding medical reports and/or other natural language textdata associated with a medical scan can be utilized to train a medicalscan natural language analysis function by generating a medical reportnatural language model. The medical scan natural language analysisfunction can be utilized to generate inference data for incoming medicalreports for other medical scans to automatically determine correspondingmedical codes, which can be mapped to corresponding medical scans.Medical codes assigned to medical scans by utilizing the medical reportnatural language model can be utilized by other subsystems, for example,to train other medical scan analysis functions, to be used as truth datato verify annotations provided via other subsystems, to aid indiagnosis, or otherwise be used by other subsystems as described herein.

A medical scan comparison system 116 can be utilized by one or moresubsystems to identify and/or display similar medical scans, forexample, to perform or determine function parameters for a medical scansimilarity analysis function, to generate or retrieve similar scan data,or otherwise compare medical scan data. The medical scan comparisonsystem 116 can also utilize some or all features of other subsystems asdescribed herein. The medical scan comparison system 116 can be operableto receive a medical scan via a network and can generate similar scandata. The similar scan data can include a subset of medical scans from amedical scan database and can be generated by performing an abnormalitysimilarity function, such as medical scan similarity analysis function,to determine that a set of abnormalities included in the subset ofmedical scans compare favorably to an abnormality identified in themedical scan. At least one cross-sectional image can be selected fromeach medical scan of the subset of medical scans for display on adisplay device associated with a user of the medical scan comparisonsystem in conjunction with the medical scan.

FIG. 2A presents an embodiment of client device 120. Each client device120 can include one or more client processing devices 230, one or moreclient memory devices 240, one or more client input devices 250, one ormore client network interfaces 260 operable to more support one or morecommunication links via the network 150 indirectly and/or directly,and/or one or more client display devices 270, connected via bus 280.Client applications 202, 204, 206, 208, 210, 212, 214, and/or 216correspond to subsystems 102, 104, 106, 108, 110, 112, 114, and/or 116of the medical scan processing system respectfully. Each client device120 can receive the application data from the corresponding subsystemvia network 150 by utilizing network interface 260, for storage in theone or more memory devices 240. In various embodiments, some or allclient devices 120 can include a computing device associated with aradiologist, medical entity, or other user of one or more subsystems asdescribed herein.

The one or more processing devices 230 can display interactive interface275 on the one or more client display devices 270 in accordance with oneor more of the client applications 202, 204, 206, 208, 210, 212, 214,and/or 216, for example, where a different interactive interface 275 isdisplayed for some or all of the client applications in accordance withthe website presented by the corresponding subsystem 102, 104, 106, 108,110, 112, 114 and/or 116. The user can provide input in response to menudata or other prompts presented by the interactive interface via the oneor more client input devices 250, which can include a microphone, mouse,keyboard, touchscreen of display device 270 itself or other touchscreen,and/or other device allowing the user to interact with the interactiveinterface. The one or more processing devices 230 can process the inputdata and/or send raw or processed input data to the correspondingsubsystem, and/or can receive and/or generate new data in response forpresentation via the interactive interface 275 accordingly, by utilizingnetwork interface 260 to communicate bidirectionally with one or moresubsystems and/or databases of the medical scan processing system vianetwork 150.

FIG. 2B presents an embodiment of a subsystem 101, which can be utilizedin conjunction with subsystem 102, 104, 106, 108, 110, 112, 114 and/or116. Each subsystem 101 can include one or more subsystem processingdevices 235, one or more subsystem memory devices 245, and/or one ormore subsystem network interfaces 265, connected via bus 285. Thesubsystem memory devices 245 can store executable instructions that,when executed by the one or more subsystem processing devices 235,facilitate performance of operations by the subsystem 101, as describedfor each subsystem herein.

FIG. 3 presents an embodiment of the database storage system 140.Database storage system 140 can include at least one database processingdevice 330, at least one database memory device 340, and at least onedatabase network interface 360, operable to more support one or morecommunication links via the network 150 indirectly and/or directly, allconnected via bus 380. The database storage system 140 can store one ormore databases the at least one memory 340, which can include a medicalscan database 342 that includes a plurality medical scan entries 352, auser database 344 that includes a plurality of user profile entries 354,a medical scan analysis function database 346 that includes a pluralityof medical scan analysis function entries 356, an interface featuredatabase 348 can include a plurality of interface feature entries 358,and/or other databases that store data generated and/or utilized by thesubsystems 101. Some or all of the databases 342, 344, 346 and/or 348can consist of multiple databases, can be stored relationally ornon-relationally, and can include different types of entries anddifferent mappings than those described herein. A database entry caninclude an entry in a relational table or entry in a non-relationalstructure. Some or all of the data attributes of an entry 352, 354, 356,and/or 358 can refer to data included in the entry itself or that isotherwise mapped to an identifier included in the entry and can beretrieved from, added to, modified, or deleted from the database storagesystem 140 based on a given identifier of the entry. Some or all of thedatabases 342, 344, 346, and/or 348 can instead be stored locally by acorresponding subsystem, for example, if they are utilized by only onesubsystem.

The processing device 330 can facilitate read/write requests receivedfrom subsystems and/or client devices via the network 150 based onread/write permissions for each database stored in the at least onememory device 340. Different subsystems can be assigned differentread/write permissions for each database based on the functions of thesubsystem, and different client devices 120 can be assigned differentread/write permissions for each database. One or more client devices 120can correspond to one or more administrators of one or more of thedatabases stored by the database storage system, and databaseadministrator devices can manage one or more assigned databases,supervise assess and/or efficiency, edit permissions, or otherwiseoversee database processes based on input to the client device viainteractive interface 275.

FIG. 4A presents an embodiment of a medical scan entry 352, stored inmedical scan database 342, included in metadata of a medical scan,and/or otherwise associated with a medical scan. A medical scan caninclude imaging data corresponding to a CT scan, x-ray, MM, PET scan,Ultrasound, EEG, mammogram, or other type of radiological scan ormedical scan taken of an anatomical region of a human body, animal,organism, or object and further can include metadata corresponding tothe imaging data. Some or all of the medical scan entries can beformatted in accordance with a Digital Imaging and Communications inMedicine (DICOM) format or other standardized image format, and some ormore of the fields of the medical scan entry 352 can be included in aDICOM header or other standardized header of the medical scan. Medicalscans can be awaiting review or can have already been reviewed by one ormore users or automatic processes and can include tentative diagnosisdata automatically generated by a subsystem, generated based on userinput, and/or generated from another source. Some medical scans caninclude final, known diagnosis data generated by a subsystem and/orgenerated based on user input, and/or generated from another source, andcan included in training sets used to train processes used by one ormore subsystems such as the medical scan image analysis system 112and/or the medical scan natural language analysis system 114.

Some medical scans can include one or more abnormalities, which can beidentified by a user or can be identified automatically. Abnormalitiescan include nodules, for example malignant nodules identified in a chestCT scan. Abnormalities can also include and/or be characterized by oneor more abnormality pattern categories such as such as cardiomegaly,consolidation, effusion, emphysema, and/or fracture, for exampleidentified in a chest x-ray. Abnormalities can also include any otherunknown, malignant or benign feature of a medical scan identified as notnormal. Some scans can contain zero abnormalities, and can be identifiedas normal scans. Some scans identified as normal scans can includeidentified abnormalities that are classified as benign, and include zeroabnormalities classified as either unknown or malignant. Scansidentified as normal scans may include abnormalities that were notdetected by one or more subsystems and/or by an originating entity.Thus, some scans may be improperly identified as normal. Similarly,scans identified to include at least one abnormality may include atleast one abnormality that was improperly detected as an abnormality byone or more subsystems and/or by an originating entity. Thus, some scansmay be improperly identified as containing abnormalities.

Each medical scan entry 352 can be identified by its own medical scanidentifier 353, and can include or otherwise map to medical scan imagedata 410, and metadata such as scan classifier data 420, patient historydata 430, diagnosis data 440, annotation author data 450, confidencescore data 460, display parameter data 470, similar scan data 480,training set data 490, and/or other data relating to the medical scan.Some or all of the data included in a medical scan entry 352 can be usedto aid a user in generating or editing diagnosis data 440, for example,in conjunction with the medical scan assisted review system 102, themedical scan report labeling system 104, and/or the medical scanannotator system 106. Some or all of the data included in a medical scanentry 352 can be used to allow one or more subsystems 101, such asautomated portions of the medical scan report labeling system 104 and/orthe medical scan diagnosing system 108, to automatically generate and/oredit diagnosis data 440 or other data the medical scan. Some or all ofthe data included in a medical scan entry 352 can be used to train someor all medical scan analysis functions of the medical scan analysisfunction database 346 such as one or more medical scan image analysisfunctions, one or more medical scan natural language analysis functions,one or more medical scan similarity analysis functions, one or moremedical report generator functions, and/or one or more medical reportanalysis functions, for example, in conjunction with the medical scanimage analysis system 112, the medical scan natural language analysissystem 114, and/or the medical scan comparison system 116.

The medical scan entries 352 and the associated data as described hereincan also refer to data associated with a medical scan that is not storedby the medical scan database, for example, that is uploaded by a clientdevice for direct transmission to a subsystem, data generated by asubsystem and used as input to another subsystem or transmitted directlyto a client device, data stored by a Picture Archive and CommunicationSystem (PACS) communicating with the medical scan processing system 100,or other data associated with a medical scan that is received and orgenerated without being stored in the medical scan database 342. Forexample, some or all of the structure and data attributes described withrespect to a medical scan entry 352 can also correspond to structureand/or data attribute of data objects or other data generated by and/ortransmitted between subsystems and/or client devices that correspond toa medical scan. Herein, any of the data attributes described withrespect to a medical scan entry 352 can also correspond to dataextracted from a data object generated by a subsystem or client deviceor data otherwise received from a subsystem, client device, or othersource via network 150 that corresponds to a medical scan.

The medical scan image data 410 can include one or more imagescorresponding to a medical scan. The medical scan image data 410 caninclude one or more image slices 412, for example, corresponding to asingle x-ray image, a plurality of cross-sectional, tomographic imagesof a scan such as a CT scan, or any plurality of images taken from thesame or different point at the same or different angles. The medicalscan image data 410 can also indicate an ordering of the one or moreimage slices 412. Herein, a “medical scan” can refer a full scan of anytype represented by medical scan image data 410. Herein, an “imageslice” can refer to one of a plurality of cross-sectional images of themedical scan image data 410, one of a plurality of images taken fromdifferent angles of the medical scan image data 410, and/or the singleimage of the medical scan image data 410 that includes only one image.Furthermore “plurality of image slices” can refer to all of the imagesof the associated medical scan, and refers to only a single image if themedical scan image data 410 includes only one image. Each image slice412 can include a plurality of pixel values 414 mapped to each pixel ofthe image slice. Each pixel value can correspond to a density value,such as a Hounsfield value or other measure of density. Pixel values canalso correspond to a grayscale value, a RGB (Red-Green-Blue) or othercolor value, or other data stored by each pixel of an image slice 412.

Scan classifier data 420 can indicate classifying data of the medicalscan. Scan classifier data can include scan type data 421, for example,indicating the modality of the scan. The scan classifier data canindicate that the scan is a CT scan, x-ray, MM, PET scan, Ultrasound,EEG, mammogram, or other type of scan. Scan classifier data 420 can alsoinclude anatomical region data 422, indicating for example, the scan isa scan of the chest, head, right knee, or other anatomical region. Scanclassifier data can also include originating entity data 423, indicatingthe hospital where the scan was taken and/or a user that uploaded thescan to the system. If the originating entity data corresponds to a userof one or more subsystems 101, the originating entity data can include acorresponding user profile identifier and/or include other data from theuser profile entry 354 of the user. Scan classifier data 420 can includegeographic region data 424, indicating a city, state, and/or countryfrom which the scan originated, for example, based on the user dataretrieved from the user database 344 based on the originating entity.Scan classifier data can also include machine data 425, which caninclude machine identifier data, machine model data, machine calibrationdata, and/or contrast agent data, for example based on imaging machinedata retrieved from the user database 344 based on the originatingentity data 423. The scan classifier data 420 can include scan date data426 indicating when the scan was taken. The scan classifier data 420 caninclude scan priority data 427, which can indicate a priority score,ranking, number in a queue, or other priority data with regard totriaging and/or review. A priority score, ranking, or queue number ofthe scan priority data 427 can be generated by automatically by asubsystem based on the scan priority data 427, based on a severity ofpatient symptoms or other indicators in the risk factor data 432, basedon a priority corresponding to the originating entity, based onpreviously generated diagnosis data 440 for the scan, and/or can beassigned by the originating entity and/or a user of the system.

The scan classifier data 420 can include other classifying data notpictured in FIG. 4A. For example, a set of scans can include medicalscan image data 410 corresponding to different imaging planes. The scanclassifier data can further include imaging plane data indicating one ormore imaging planes corresponding to the image data. For example, theimaging plane data can indicate the scan corresponds to the axial plane,sagittal plane, or coronal plane. A single medical scan entry 352 caninclude medical scan image data 410 corresponding multiple planes, andeach of these planes can be tagged appropriately in the image data. Inother embodiments, medical scan image data 410 corresponding to eachplane can be stored as separate medical scan entries 352, for example,with a common identifier indicating these entries belong to the same setof scans.

Alternatively or in addition, the scan classifier data 420 can includesequencing data. For example, a set of scans can include medical scanimage data 410 corresponding to different sequences. The scan classifierdata can further include sequencing data indicating one or more of aplurality of sequences of the image data corresponds to, for example,indicating whether an MRI scan corresponds to a T2 sequence, a T1sequence, a T1 sequence with contrast, a diffusion sequence, a FLAIRsequence, or other MRI sequence. A single medical scan entry 352 caninclude medical scan image data 410 corresponding to multiple sequences,and each of these sequences can be tagged appropriately in the entry. Inother embodiments, medical scan image data 410 corresponding to eachsequence can be stored as separate medical scan entries 352, forexample, with a common identifier indicating these entries belong to thesame set of scans.

Alternatively or in addition, the scan classifier data 420 can includean image quality score. This score can be determined automatically byone or more subsystems 101, and/or can be manually assigned the medicalscan. The image quality score can be based on a resolution of the imagedata 410, where higher resolution image data is assigned a morefavorable image quality score than lower resolution image data. Theimage quality score can be based on whether the image data 410corresponds to digitized image data received directly from thecorresponding imaging machine, or corresponds to a hard copy of theimage data that was later scanned in. In some embodiments, the imagequality score can be based on a detected corruption, and/or detectedexternal factor that determined to negatively affect the quality of theimage data during the capturing of the medical scan and/or subsequent tothe capturing of the medical scan. In some embodiments, the imagequality score can be based on detected noise in the image data, where amedical scan with a higher level of detected noise can receive a lessfavorable image quality score than a medical scan with a lower level ofdetected noise. Medical scans with this determined corruption orexternal factor can receive a less favorable image quality score thanmedical scans with no detected corruption or external factor.

In some embodiments, the image quality score can be based on includemachine data 425. In some embodiments, one or more subsystems canutilize the image quality score to flag medical scans with image qualityscores that fall below an image quality threshold. The image qualitythreshold can be the same or different for different subsystems, medicalscan modalities, and/or anatomical regions. For example, the medicalscan image analysis system can automatically filter training sets basedon selecting only medical scans with image quality scores that comparefavorably to the image quality threshold. As another example, one ormore subsystems can flag a particular imaging machine and/or hospital orother medical entity that have produced at least a threshold numberand/or percentage of medical scan with image quality scores that compareunfavorably to the image quality threshold. As another example, ade-noising algorithm can be automatically utilized to clean the imagedata when the image quality score compares unfavorably to the imagequality threshold. As another example, the medical scan image analysissystem can select a particular medical image analysis function from aset of medical image analysis functions to utilize on a medical scan togenerate inference data for the medical scan. Each of this set ofmedical image analysis function can be trained on different levels ofimage quality, and the selected image analysis function can be selectedbased on the determined image quality score falling within a range ofimage quality scores the image analysis function was trained on and/oris otherwise suitable for.

The patient history data 430 can include patient identifier data 431which can include basic patient information such as name or anidentifier that may be anonymized to protect the confidentiality of thepatient, age, and/or gender. The patient identifier data 431 can alsomap to a patient entry in a separate patient database stored by thedatabase storage system, or stored elsewhere. The patient history datacan include patient risk factor data 432 which can include previousmedical history, family medical history, smoking and/or drug habits,pack years corresponding to tobacco use, environmental exposures,patient symptoms, etc. The patient history data 430 can also includelongitudinal data 433, which can identify one or more additional medicalscans corresponding to the patient, for example, retrieved based onpatient identifier data 431 or otherwise mapped to the patientidentifier data 431. Some or all additional medical scans can beincluded in the medical scan database, and can be identified based ontheir corresponding identifiers medical scan identifiers 353. Some orall additional medical scans can be received from a different source andcan otherwise be identified. Alternatively or in addition, thelongitudinal data can simply include some or all relevant scan entrydata of a medical scan entry 352 corresponding to the one or moreadditional medical scans. The additional medical scans can be the sametype of scan or different types of scans. Some or all of the additionalscans may correspond to past medical scans, and/or some or all of theadditional scans may correspond to future medical scans. Thelongitudinal data 433 can also include data received and/or determinedat a date after the scan such as final biopsy data, or some or all ofthe diagnosis data 440. The patient history data can also include alongitudinal quality score 434, which can be calculated automatically bya subsystem, for example, based on the number of additional medicalscans, based on how many of the additional scans in the file were takenbefore and/or after the scan based on the scan date data 426 of themedical scan and the additional medical scans, based on a date rangecorresponding to the earliest scan and corresponding to the latest scan,based on the scan types data 421 these scans, and/or based on whether ornot a biopsy or other final data is included. As used herein, a “high”longitudinal quality score refers to a scan having more favorablelongitudinal data than that with a “low” longitudinal quality score.

Diagnosis data 440 can include data that indicates an automateddiagnosis, a tentative diagnosis, and/or data that can otherwise be usedto support medical diagnosis, triage, medical evaluation and/or otherreview by a medical professional or other user. The diagnosis data 440of a medical scan can include a binary abnormality identifier 441indicating whether the scan is normal or includes at least oneabnormality. In some embodiments, the binary abnormality identifier 441can be determined by comparing some or all of confidence score data 460to a threshold, can be determined by comparing a probability value to athreshold, and/or can be determined by comparing another continuous ordiscrete value indicating a calculated likelihood that the scan containsone or more abnormalities to a threshold. In some embodiments,non-binary values, such as one or more continuous or discrete valuesindicating a likelihood that the scan contains one or moreabnormalities, can be included in diagnosis data 440 in addition to, orinstead of, binary abnormality identifier 441. One or abnormalities canbe identified by the diagnosis data 440, and each identified abnormalitycan include its own set of abnormality annotation data 442.Alternatively, some or all of the diagnosis data 440 can indicate and/ordescribe multiple abnormalities, and thus will not be presented for eachabnormality in the abnormality annotation data 442. For example, thereport data 449 of the diagnosis data 440 can describe all identifiedabnormalities, and thus a single report can be included in thediagnosis.

FIG. 4B presents an embodiment of the abnormality annotation data 442.The abnormality annotation data 442 for each abnormality can includeabnormality location data 443, which can include an anatomical locationand/or a location specific to pixels, image slices, coordinates or otherlocation information identifying regions of the medical scan itself. Theabnormality annotation data 442 can include abnormality classificationdata 445 which can include binary, quantitative, and/or descriptive dataof the abnormality as a whole, or can correspond to one or moreabnormality classifier categories 444, which can include size, volume,pre-post contrast, doubling time, calcification, components, smoothness,spiculation, lobulation, sphericity, internal structure, texture, orother categories that can classify and/or otherwise characterize anabnormality. Abnormality classifier categories 444 can be assigned abinary value, indicating whether or not such a category is present. Forexample, this binary value can be determined by comparing some or all ofconfidence score data 460 to a threshold, can be determined by comparinga probability value to a threshold, and/or can be determined bycomparing another continuous or discrete value indicating a calculatedlikelihood that a corresponding abnormality classifier category 444 ispresent to a threshold, which can be the same or different threshold foreach abnormality classifier category 444. In some embodiments,abnormality classifier categories 444 can be assigned one or morenon-binary values, such as one or more continuous or discrete valuesindicating a likelihood that the corresponding classifier category 444is present.

The abnormality classifier categories 444 can also include a malignancycategory, and the abnormality classification data 445 can include amalignancy rating such as a Lung-RADS score, a Fleischner score, and/orone or more calculated values that indicate malignancy level, malignancyseverity, and/or probability of malignancy. Alternatively or inaddition, the malignancy category can be assigned a value of “yes”,“no”, or “maybe”. The abnormality classifier categories 444 can alsoinclude abnormality pattern categories 446 such as cardiomegaly,consolidation, effusion, emphysema, and/or fracture, and the abnormalityclassification data 445 for each abnormality pattern category 446 canindicate whether or not each of the abnormality patterns is present.

The abnormality classifier categories can correspond to ResponseEvaluation Criteria in Solid Tumors (RECIST) eligibility and/or RECISTevaluation categories. For example, an abnormality classifier category444 corresponding to RECIST eligibility can have correspondingabnormality classification data 445 indicating a binary value “yes” or“no”, and/or can indicate if the abnormality is a “target lesion” and/ora “non-target lesion.” As another example, an abnormality classifiercategory 444 corresponding to a RECIST evaluation category can bedetermined based on longitudinal data 433 and can have correspondingabnormality classification data 445 that includes one of the set ofpossible values “Complete Response”, “Partial Response”, “StableDisease”, or “Progressive Disease.”

The diagnosis data 440 as a whole, and/or the abnormality annotationdata 442 for each abnormality, can include custom codes or datatypesidentifying the binary abnormality identifier 441, abnormality locationdata 443 and/or some or all of the abnormality classification data 445of one or more abnormality classifier categories 444. Alternatively orin addition, some or all of the abnormality annotation data 442 for eachabnormality and/or other diagnosis data 440 can be presented in a DICOMformat or other standardized image annotation format, and/or can beextracted into custom datatypes based on abnormality annotation dataoriginally presented in DICOM format. Alternatively or in addition, thediagnosis data 440 and/or the abnormality annotation data 442 for eachabnormality can be presented as one or more medical codes 447 such asSNOMED codes, Current Procedure Technology (CPT) codes, ICD-9 codes,ICD-10 codes, or other standardized medical codes used to label orotherwise describe medical scans.

Alternatively or in addition, the diagnosis data 440 can include naturallanguage text data 448 annotating or otherwise describing the medicalscan as a whole, and/or the abnormality annotation data 442 can includenatural language text data 448 annotating or otherwise describing eachcorresponding abnormality. In some embodiments, some or all of thediagnosis data 440 is presented only as natural language text data 448.In some embodiments, some or all of the diagnosis data 440 isautomatically generated by one or more subsystems based on the naturallanguage text data 448, for example, without utilizing the medical scanimage data 410, for example, by utilizing one or more medical scannatural language analysis functions trained by the medical scan naturallanguage analysis system 114. Alternatively or in addition, someembodiments, some or all of the natural language text data 448 isgenerated automatically based on other diagnosis data 440 such asabnormality annotation data 442, for example, by utilizing a medicalscan natural language generating function trained by the medical scannatural language analysis system 114.

The diagnosis data can include report data 449 that includes at leastone medical report, which can be formatted to include some or all of themedical codes 447, some or all of the natural language text data 448,other diagnosis data 440, full or cropped images slices formatted basedon the display parameter data 470 and/or links thereto, full or croppedimages slices or other data based on similar scans of the similar scandata 480 and/or links thereto, full or cropped images or other databased on patient history data 430 such as longitudinal data 433 and/orlinks thereto, and/or other data or links to data describing the medicalscan and associated abnormalities. The diagnosis data 440 can alsoinclude finalized diagnosis data corresponding to future scans and/orfuture diagnosis for the patient, for example, biopsy data or otherlongitudinal data 433 determined subsequently after the scan. Themedical report of report data 449 can be formatted based on specifiedformatting parameters such as font, text size, header data, bulleting ornumbering type, margins, file type, preferences for including one ormore full or cropped image slices 412, preferences for including similarmedical scans, preferences for including additional medical scans, orother formatting to list natural language text data and/or image data,for example, based on preferences of a user indicated in the originatingentity data 423 or other responsible user in the corresponding reportformatting data.

Annotation author data 450 can be mapped to the diagnosis data for eachabnormality, and/or mapped to the scan as a whole. This can include oneor more annotation author identifiers 451, which can include one or moreuser profile identifiers of a user of the system, such as an individualmedical professional, medical facility and/or medical entity that usesthe system. Annotation author data 450 can be used to determine theusage data of a user profile entry 354. Annotation author data 450 canalso include one or more medical scan analysis function identifiers 357or other function identifier indicating one or more functions or otherprocesses of a subsystem responsible for automatically generating and/orassisting a user in generating some or all of the diagnosis data, forexample an identifier of a particular type and/or version of a medicalscan image analysis functions that was used by the medical scandiagnosing system 108 used to generate part or all of the diagnosis data440 and/or an interface feature identifier, indicating an one or moreinterface features presented to a user to facilitate entry of and/orreviewing of the diagnosis data 440. The annotation author data can alsosimply indicate, for one or more portions of the diagnosis data 440, ifthis portion was generated by a human or automatically generated by asubsystem of the medical scan processing system.

In some embodiments, if a medical scan was reviewed by multipleentities, multiple, separate diagnosis data entries 440 can be includedin the medical scan entry 352, mapped to each diagnosis author in theannotation author data 450. This allows different versions of diagnosisdata 440 received from multiple entities. For example, annotation authordata of a particular medical scan could indicate that the annotationdata was written by a doctor at medical entity A, and the medical codedata was generated by user Y by utilizing the medical scan reportlabeling system 104, which was confirmed by expert user X. Theannotation author data of another medical scan could indicate that themedical code was generated automatically by utilizing version 7 of themedical scan image analysis function relating to chest x-rays, andconfirmed by expert user X. The annotation author data of anothermedical scan could indicate that the location and a first malignancyrating were generated automatically by utilizing version 7 of themedical scan image analysis function relating to chest x-rays, and thata second malignancy rating was entered by user Z. In some embodiments,one of the multiple diagnosis entries can include consensus annotationdata, for example, generated automatically by a subsystem such as themedical scan annotating system 106 based on the multiple diagnosis data440, based on confidence score data 460 of each of the multiplediagnosis data 440, and/or based on performance score data of acorresponding user, a medical scan analysis function, or an interfacefeature, identified in the annotation author data for each correspondingone of the multiple diagnosis data 440.

Confidence score data 460 can be mapped to some or all of the diagnosisdata 440 for each abnormality, and/or for the scan as a whole. This caninclude an overall confidence score for the diagnosis, a confidencescore for the binary indicator of whether or not the scan was normal, aconfidence score for the location a detected abnormality, and/orconfidence scores for some or all of the abnormality classifier data.This may be generated automatically by a subsystem, for example, basedon the annotation author data and corresponding performance score of oneor more identified users and/or subsystem attributes such as interactiveinterface types or medical scan image analysis functions indicated bythe annotation author data. In the case where multiple diagnosis dataentries 440 are included from different sources, confidence score data460 can be computed for each entry and/or an overall confidence score,for example, corresponding to consensus diagnosis data, can be based oncalculated distance or other error and/or discrepancies between theentries, and/or can be weighted on the confidence score data 460 of eachentry. In various embodiments, the confidence score data 460 can includea truth flag 461 indicating the diagnosis data is considered as “known”or “truth”, for example, flagged based on user input, flaggedautomatically based on the author data, and/or flagged automaticallybased on the calculated confidence score of the confidence score dataexceeding a truth threshold. As used herein, a “high” confidence scorerefers to a greater degree or more favorable level of confidence than a“low” confidence score.

Display parameter data 470 can indicate parameters indicating an optimalor preferred display of the medical scan by an interactive interface 275and/or formatted report for each abnormality and/or for the scan as awhole. Some or all of the display parameter data can have separateentries for each abnormality, for example, generated automatically by asubsystem 101 based on the abnormality annotation data 442. Displayparameter data 470 can include interactive interface feature data 471,which can indicate one or more selected interface features associatedwith the display of abnormalities and/or display of the medical scan asa whole, and/or selected interface features associated with userinteraction with a medical scan, for example, based on categorizedinterface feature performance score data and a category associated withthe abnormality and/or with the medical scan itself. The displayparameter data can include a slice subset 472, which can indicate aselected subset of the plurality of image slices that includes a singleimage slice 412 or multiple image slices 412 of the medical scan imagedata 410 for display by a user interface. The display parameter data 470can include slice order data 473 that indicates a selected customordering and/or ranking for the slice subset 472, or for all of theslices 412 of the medical scan. The display parameter data 470 caninclude slice cropping data 474 corresponding to some or all of theslice subset 472, or all of the image slices 412 of the medical scan,and can indicating a selected custom cropped region of each image slice412 for display, or the same selected custom cropped region for theslice subset 472 or for all slices 412. The display parameter data caninclude density window data 475, which can indicate a selected customdensity window for display of the medical scan as a whole, a selectedcustom density window for the slice subset 472, and/or selected customdensity windows for each of the image slices 412 of the slice subset472, and/or for each image slice 412 of the medical scan. The densitywindow data 475 can indicate a selected upper density value cut off anda selected lower density value cut off, and/or can include a selecteddeterministic function to map each density value of a pixel to agrayscale value based on the preferred density window. The interactiveinterface feature data 471, slice subset 472, slice order data 473,slice cropping data 474, and/or the density window data 475 can beselected via user input and/or generated automatically by one or moresubsystems 101, for example, based on the abnormality annotation data442 and/or based on performance score data of different interactiveinterface versions.

Similar scan data 480 can be mapped to each abnormality, or the scan asa whole, and can include similar scan identifier data 481 correspondingto one or more identified similar medical scans, for example,automatically identified by a subsystem 101, for example, by applying asimilar scan identification step of the medical scan image analysissystem 112 and/or applying medical scan similarity analysis function tosome or all of the data stored in the medical scan entry of the medicalscan, and/or to some or all corresponding data of other medical scans inthe medical scan database. The similar scan data 480 can also correspondto medical scans received from another source. The stored similaritydata can be used to present similar cases to users of the system and/orcan be used to train medical scan image analysis functions or medicalscan similarity analysis functions.

Each identified similar medical scan can have its own medical scan entry352 in the medical scan database 342 with its own data, and the similarscan identifier data 481 can include the medical scan identifier 353each similar medical scan. Each identified similar medical scan can be ascan of the same scan type or different scan type than medical scan.

The similar scan data 480 can include a similarity score 482 for eachidentified similar scan, for example, generated based on some or all ofthe data of the medical scan entry 352 for medical scan and based onsome or all of the corresponding data of the medical scan entry 352 forthe identified similar medical scan. For example, the similarity score482 can be generated based on applying a medical scan similarityanalysis function to the medical image scan data of medical scans and402, to some or all of the abnormality annotation data of medical scansand 402, and/or to some or all of the patient history data 430 ofmedical scans and 402 such as risk factor data 432. As used herein, a“high” similarity score refers a higher level of similarity that a “low”similarity score.

The similar scan data 480 can include its own similar scan displayparameter data 483, which can be determined based on some or all of thedisplay parameter data 470 of the identified similar medical scan. Someor all of the similar scan display parameter data 483 can be generatedautomatically by a subsystem, for example, based on the displayparameter data 470 of the identified similar medical scan, based on theabnormality annotation data 442 of the medical scan itself and/or basedon display parameter data 470 of the medical scan itself. Thus, thesimilar scan display parameter data 483 can be the same or differentthan the display parameter data 470 mapped to the identified similarmedical scan and/or can be the same or different than the displayparameter data 470 of the medical scan itself. This can be utilized whendisplaying similar scans to a user via interactive interface 275 and/orcan be utilized when generating report data 449 that includes similarscans, for example, in conjunction with the medical scan assisted reviewsystem 102.

The similar scan data 480 can include similar scan abnormality data 484,which can indicate one of a plurality of abnormalities of the identifiedsimilar medical scan and its corresponding abnormality annotation data442. For example, the similarity scan abnormality data 484 can includean abnormality pair that indicates one of a plurality of abnormalitiesof the medical scan, and indicates one of a plurality of abnormalitiesof the identified similar medical scan, for example, that was identifiedas the similar abnormality.

The similar scan data 480 can include similar scan filter data 485. Thesimilar scan filter data can be generated automatically by a subsystem,and can include a selected ordered or un-ordered subset of allidentified similar scans of the similar scan data 480, and/or a rankingof all identified similar scans. For example, the subset can be selectedand/or some or all identified similar scans can be ranked based on eachsimilarity score 482, and/or based on other factors such as based on alongitudinal quality score 434 of each identified similar medical scan.

The training set data 490 can indicate one or more training sets thatthe medical scan belongs to. For example, the training set data canindicate one or more training set identifiers 491 indicating one or moremedical scan analysis functions that utilized the medical scan in theirtraining set, and/or indicating a particular version identifier 641 ofthe one or more medical scan analysis functions that utilized themedical scan in their training set. The training set data 490 can alsoindicate which portions of the medical scan entry were utilized by thetraining set, for example, based on model parameter data 623 of thecorresponding medical scan analysis functions. For example, the trainingset data 490 can indicate that the medical scan image data 410 wasincluded in the training set utilized to train version X of the chestx-ray medical scan image analysis function, or that the natural languagetext data 448 of this medical scan was used to train version Y of thenatural language analysis function.

FIG. 5A presents an embodiment of a user profile entry 354, stored inuser database 344 or otherwise associated with a user. A user cancorrespond to a user of one or more of the subsystems such as aradiologist, doctor, medical professional, medical report labeler,administrator of one or more subsystems or databases, or other user thatuses one or more subsystems 101. A user can also correspond to a medicalentity such as a hospital, medical clinic, establishment that utilizesmedical scans, establishment that employs one or more of the medicalprofessionals described, an establishment associated with administeringone or more subsystems, or other entity. A user can also correspond to aparticular client device 120 or account that can be accessed one or moremedical professionals or other employees at the same or differentmedical entities. Each user profile entry can have a corresponding userprofile identifier 355.

A user profile entry 354 can include basic user data 510, which caninclude identifying information 511 corresponding to the user such as aname, contact information, account/login/password information,geographic location information such as geographic region data 424,and/or other basic information. Basic user data 510 can includeaffiliation data 512, which can list one or more medical entities orother establishments the user is affiliated with, for example, if theuser corresponds to a single person such as a medical professional, orif the user corresponds to a hospital in a network of hospitals. Theaffiliation data 512 can include one or more corresponding user profileidentifiers 355 and/or basic user data 510 if the correspondingaffiliated medical entity or other establishment has its own entry inthe user database. The user identifier data can include employee data513 listing one or more employees, such as medical professionals withtheir own user profile entries 354, for example, if the user correspondsto a medical entity or supervising medical professional of other medicalprofessional employees, and can list a user profile identifier 355and/or basic user data 510 for each employee. The basic user data 510can also include imaging machine data 514, which can include a list ofmachines affiliated with the user which can include machine identifiers,model information, calibration information, scan type information, orother data corresponding to each machine, for example, corresponding tothe machine data 425. The user profile entry can include client devicedata 515, which can include identifiers for one or more client devicesassociated with the user, for example, allowing subsystems 101 to senddata to a client device 120 corresponding to a selected user based onthe client device data and/or to determine a user that data was receivedby determining the client device from which the data was received.

The user profile entry can include usage data 520 which can includeidentifying information for a plurality of usages by the user inconjunction with using one or more subsystems 101. This can includeconsumption usage data 521, which can include a listing of, or aggregatedata associated with, usages of one or more subsystems by the user, forexample, where the user is utilizing the subsystem as a service. Forexample, the consumption usage data 521 can correspond to each instancewhere diagnosis data was sent to the user for medical scans provided tothe user in conjunction with the medical scan diagnosing system 108and/or the medical scan assisted review system 102. Some or all ofconsumption usage data 521 can include training usage data 522,corresponding to usage in conjunction with a certification program orother user training provided by one or more subsystems. The trainingusage data 522 can correspond to each instance where diagnosis feedbackdata was provided by user for a medical scan with known diagnosis data,but diagnosis feedback data is not utilized by a subsystem to generate,edit, and/or confirm diagnosis data 440 of the medical scan, as it isinstead utilized to train a user and/or determine performance data for auser.

Usage data 520 can include contribution usage data 523, which caninclude a listing of, or aggregate data associated with, usages of oneor more subsystems 101 by the user, for example, where the user isgenerating and/or otherwise providing data and/or feedback that can isutilized by the subsystems, for example, to generate, edit, and/orconfirm diagnosis data 440 and/or to otherwise populate, modify, orconfirm portions of the medical scan database or other subsystem data.For example, the contribution usage data 523 can correspond to diagnosisfeedback data received from user, used to generate, edit, and/or confirmdiagnosis data. The contribution usage data 523 can include interactiveinterface feature data 524 corresponding to the interactive interfacefeatures utilized with respect to the contribution.

The consumption usage data 521 and/or the contribution usage data 523can include medical scan entry 352 whose entries the user utilizedand/or contributed to, can indicate one or more specific attributes of amedical scan entry 352 that a user utilized and/or contributed to,and/or a log of the user input generated by a client device of the userin conjunction with the data usage. The contribution usage data 523 caninclude the diagnosis data that the user may have generated and/orreviewed, for example, indicated by, mapped to, and/or used to generatethe annotation author data 450 of corresponding medical scan entries352. Some usages may correspond to both consumption usage of theconsumption usage data 521 and contribution usage of the contributionusage data 523. The usage data 520 can also indicate one or moresubsystems 101 that correspond to each consumption and/or contribution.

The user profile entry can include performance score data 530. This caninclude one or more performance scores generated based on thecontribution usage data 523 and/or training usage data 522. Theperformance scores can include separate performance scores generated forevery contribution in the contribution usage data 523 and/or trainingusage data 522 and/or generated for every training consumption usagescorresponding to a training program. As used herein, a “high”performance score refers to a more favorable performance or rating thana “low” performance score.

The performance score data can include accuracy score data 531, whichcan be generated automatically by a subsystem for each contribution, forexample, based on comparing diagnosis data received from a user to datato known truth data such as medical scans with a truth flag 461, forexample, retrieved from the corresponding medical scan entry 352 and/orbased on other data corresponding to the medical scan, for example,received from an expert user that later reviewed the contribution usagedata of the user and/or generated automatically by a subsystem. Theaccuracy score data 531 can include an aggregate accuracy scoregenerated automatically by a subsystem, for example, based on theaccuracy data of multiple contributions by the user over time.

The performance data can also include efficiency score data 532generated automatically by a subsystem for each contribution based on anamount of time taken to complete a contribution, for example, from atime the request for a contribution was sent to the client device to atime that the contribution was received from the client device, based ontiming data received from the client device itself, and/or based onother factors. The efficiency score can include an aggregate efficiencyscore, which can be generated automatically by a subsystem based on theindividual efficiency scores over time and/or based on determining acontribution completion rate, for example based on determining how manycontributions were completed in a fixed time window.

Aggregate performance score data 533 can be generated automatically by asubsystem based on the aggregate efficiency and/or accuracy data. Theaggregate performance data can include categorized performance data 534,for example, corresponding to different scan types, different anatomicalregions, different subsystems, different interactive interface featuresand/or display parameters. The categorized performance data 534 can bedetermined automatically by a subsystem based on the scan type data 421and/or anatomical region data 422 of the medical scan associated witheach contribution, one or more subsystems 101 associated with eachcontribution, and/or interactive interface feature data 524 associatedwith each contribution. The aggregate performance data can also be basedon performance score data 530 of individual employees if the usercorresponds to a medical entity, for example, retrieved based on userprofile identifiers 355 included in the employee data 513. Theperformance score data can also include ranking data 535, which caninclude an overall ranking or categorized rankings, for example,generated automatically by a subsystem or the database itself based onthe aggregate performance data.

In some embodiments, aggregate data for each user can be further brokendown based on scores for distinct scan categories, for example, based onthe scan classifier data 420, for example, where a first aggregate datascore is generated for a user “A” based on scores from all knee x-rays,and a second aggregate data score is generated for user A based onscores from all chest CT scans. Aggregate data for each user can befurther based on scores for distinct diagnosis categories, where a firstaggregate data score is generated for user A based on scores from allnormal scans, and a second aggregate data score is generated for user Abased on scores from all scans that contain an abnormality. This can befurther broken down, where a first aggregate score is generated for userA based on all scores from scans that contain an abnormality of a firsttype and/or in a first anatomical location, and a second aggregate scoreis generated for A based on all scores from scans that contain anabnormality of a second type and/or in a second location. Aggregate datafor each user can be further based on affiliation data, where a rankingis generated for a medical professional “B” based on scores from allmedical professionals with the same affiliation data, and/or where aranking is generated for a hospital “C” based on scores for allhospitals, all hospitals in the same geographical region, etc. Aggregatedata for each user can be further based on scores for interfacefeatures, where a first aggregate data score is generated for user Abased on scores using a first interface feature, and a second aggregatedata score is generated for user A based on scores using a firstinterface feature.

The user profile entry can include qualification data 540. Thequalification data can include experience data 541 such as educationdata, professional practice data, number of years practicing, awardsreceived, etc. The qualification data 540 can also include certificationdata 542 corresponding to certifications earned based on contributionsto one or more subsystems, for example, assigned to users automaticallyby a subsystem based on the performance score data 530 and/or based on anumber of contributions in the contribution usage data 523 and/ortraining usage data 522. For example, the certifications can correspondto standard and/or recognized certifications to train medicalprofessionals and/or incentivize medical professionals to use thesystem. The qualification data 540 can include expert data 543. Theexpert data 543 can include a binary expert identifier, which can begenerated automatically by a subsystem based on experience data 541,certification data 542, and/or the performance score data 530, and canindicate whether the user is an expert user. The expert data 543 caninclude a plurality of categorized binary expert identifierscorresponding to a plurality of qualification categories correspondingto corresponding to scan types, anatomical regions, and/or theparticular subsystems. The categorized binary expert identifiers can begenerated automatically by a subsystem based on the categorizedperformance data 534 and/or the experience data 541. The categories beranked by performance score in each category to indicate particularspecialties. The expert data 543 can also include an expert ranking orcategorized expert ranking with respect to all experts in the system.

The user profile entry can include subscription data 550, which caninclude a selected one of a plurality of subscription options that theuser has subscribed to. For example, the subscription options cancorrespond to allowed usage of one or more subsystems, such as a numberof times a user can utilize a subsystem in a month, and/or to acertification program, for example paid for by a user to receivetraining to earn a subsystem certification of certification data 542.The subscription data can include subscription expiration information,and/or billing information. The subscription data can also includesubscription status data 551, which can for example indicate a number ofremaining usages of a system and/or available credit information. Forexample, the remaining number of usages can decrease and/or availablecredit can decrease in response to usages that utilize one or moresubsystems as a service, for example, indicated in the consumption usagedata 521 and/or training usage data 522. In some embodiments, theremaining number of usages can increase and/or available credit canincrease in response to usages that correspond to contributions, forexample, based on the contribution usage data 523. An increase in creditcan be variable, and can be based on a determined quality of eachcontribution, for example, based on the performance score data 530corresponding to the contribution where a higher performance scorecorresponds to a higher increase in credit, based on scan priority data427 of the medical scan where contributing to higher priority scanscorresponds to a higher increase in credit, or based on other factors.

The user profile entry 354 can include interface preference data 560.The interface preference data can include a preferred interactiveinterface feature set 561, which can include one or more interactiveinterface feature identifiers and/or one or more interactive interfaceversion identifiers of interface feature entries 358 and/or versionidentifiers of the interface features. Some or all of the interfacefeatures of the preferred interactive interface feature set 561 cancorrespond to display parameter data 470 of medical scans. The preferredinteractive interface feature set 561 can include a single interactivefeature identifier for one or more feature types and/or interface types,and/or can include a single interactive interface version identifier forone or more interface categories. The preferred interactive interfacefeature set 561 can include a ranking of multiple features for the samefeature type and/or interface type. The ranked and/or unranked preferredinteractive interface feature set 561 can be generated based on userinput to an interactive interface of the client device to select and/orrank some or all of the interface features and/or versions. Some or allof the features and/or versions of the preferred interactive feature setcan be selected and/or ranked automatically by a subsystem such as themedical scan interface evaluator system, for example based on interfacefeature performance score data and/or feature popularity data.Alternatively or in addition, the performance score data 530 can beutilized by a subsystem to automatically determine the preferredinteractive feature set, for example, based on the scores in differentfeature-based categories of the categorized performance data 534.

The user profile entry 354 can include report formatting data 570, whichcan indicate report formatting preferences indicated by the user. Thiscan include font, text size, header data, bulleting or numbering type,margins, file type, preferences for including one or more full orcropped image slices 412, preferences for including similar medicalscans, preferences for including additional medical scans in reports, orother formatting preference to list natural language text data and/orimage data corresponding to each abnormality. Some or all of the reportformatting data 570 can be based on interface preference data 560. Thereport formatting data 570 can be used by one or more subsystems toautomatically generate report data 449 of medical scans based on thepreferences of the requesting user.

FIG. 5B presents an embodiment of a medical scan analysis function entry356, stored in medical scan analysis function database 346 or otherwiseassociated with one of a plurality of medical scan analysis functionstrained by and/or utilized by one or more subsystems 101. For example, amedical scan analysis function can include one or more medical scanimage analysis functions trained by the medical scan image analysissystem 112; one or more medical scan natural language analysis functionstrained by the medical scan natural language analysis system 114; one ormore medical scan similarity analysis function trained by the medicalscan image analysis system 112, the medical scan natural languageanalysis system 114, and/or the medical scan comparison system 116; oneor more medical report generator functions trained by the medical scannatural language analysis system 114 and/or the medical scan imageanalysis system 112, and/or the medical report analysis function trainedby the medical scan natural language analysis system 114. Some or all ofthe medical scan analysis functions can correspond to medical scaninference functions of the medical scan diagnosing system 108, thede-identification function and/or the inference functions utilized by amedical picture archive integration system as discussed in conjunctionwith FIGS. 8A-8F, or other functions and/or processes described hereinin conjunction with one or more subsystems 101. Each medical scananalysis function entry 356 can include a medical scan analysis functionidentifier 357.

A medical scan analysis function entry 356 can include functionclassifier data 610. Function classifier data 610 can include input andoutput types corresponding to the function. For example the functionclassifier data can include input scan category 611 that indicates whichtypes of scans can be used as input to the medical scan analysisfunction. For example, input scan category 611 can indicate that amedical scan analysis function is for chest CT scans from a particularhospital or other medical entity. The input scan category 611 caninclude one or more categories included in scan classifier data 420. Invarious embodiments, the input scan category 611 corresponds to thetypes of medical scans that were used to train the medical scan analysisfunction. Function classifier data 610 can also include output type data612 that characterizes the type of output that will be produced by thefunction, for example, indicating that a medical scan analysis functionis used to generate medical codes 447. The input scan category 611 canalso include information identifying which subsystems 101 areresponsible for running the medical scan analysis function.

A medical scan analysis function entry 356 can include trainingparameters 620. This can include training set data 621, which caninclude identifiers for the data used to train the medical scan analysisfunction, such as a set of medical scan identifiers 353 corresponding tothe medical scans used to train the medical scan analysis function, alist of medical scan reports and corresponding medical codes used totrain the medical scan analysis function, etc. Alternatively or inaddition to identifying particular scans of the training set, thetraining set data 621 can identify training set criteria, such asnecessary scan classifier data 420, necessary abnormality locations,classifiers, or other criteria corresponding to abnormality annotationdata 442, necessary confidence score data 460, for example, indicatingthat only medical scans with diagnosis data 440 assigned a truth flag461 or with confidence score data 460 otherwise comparing favorably to atraining set confidence score threshold are included, a number ofmedical scans to be included and proportion data corresponding todifferent criteria, or other criteria used to populate a training setwith data of medical scans. Training parameters 620 can include modeltype data 622 indicating one or more types of model, methods, and/ortraining functions used to determine the medical scan analysis functionby utilizing the training set 621. Training parameters 620 can includemodel parameter data 623 that can include a set of features of thetraining data selected to train the medical scan analysis function,determined values for weights corresponding to selected input and outputfeatures, determined values for model parameters corresponding to themodel itself, etc. The training parameter data can also include testingdata 624, which can identify a test set of medical scans or other dataused to test the medical scan analysis function. The test set can be asubset of training set 621, include completely separate data thantraining set 621, and/or overlap with training set 621. Alternatively orin addition, testing data 624 can include validation parameters such asa percentage of data that will be randomly or pseudo-randomly selectedfrom the training set for testing, parameters characterizing a crossvalidation process, or other information regarding testing. Trainingparameters 620 can also include training error data 625 that indicates atraining error associated with the medical scan analysis function, forexample, based on applying cross validation indicated in testing data624.

A medical scan analysis function entry 356 can include performance scoredata 630. Performance data can include model accuracy data 631, forexample, generated and/or updated based on the accuracy of the functionwhen performed on new data. For example, the model accuracy data 631 caninclude or be calculated based on the model error for determined forindividual uses, for example, generated by comparing the output of themedical scan analysis function to corresponding data generated by userinput to interactive interface 275 in conjunction with a subsystem 101and/or generated by comparing the output of the medical scan analysisfunction to medical scans with a truth flag 461. The model accuracy data631 can include aggregate model accuracy data computed based on modelerror of individual uses of the function over time. The performancescore data 630 can also include model efficiency data 632, which can begenerated based on how quickly the medical scan analysis functionperforms, how much memory is utilized by medical scan analysis function,or other efficiency data relating to the medical scan analysis function.Some or all of the performance score data 630 can be based on trainingerror data 625 or other accuracy and/or efficiency data determinedduring training and/or validation. As used herein, a “high” performancescore refers to a more favorable performance or rating than a “low”performance score.

A medical scan analysis function entry 356 can include version data 640.The version data can include a version identifier 641. The version datacan indicate one or more previous version identifiers 642, which can mapto version identifiers 641 stored in other medical scan analysisfunction entry 356 that correspond to previous versions of the function.Alternatively or in addition, the version data can indicate multipleversions of the same type based on function classifier data 610, canindicate the corresponding order and/or rank of the versions, and/or canindicate training parameters 620 associated with each version.

A medical scan analysis function entry 356 can include remediation data650. Remediation data 650 can include remediation instruction data 651which can indicate the steps in a remediation process indicating how amedical scan analysis function is taken out of commission and/orreverted to a previous version in the case that remediation isnecessary. The version data 640 can further include remediation criteriadata 652, which can include threshold data or other criteria used toautomatically determine when remediation is necessary. For example, theremediation criteria data 652 can indicate that remediation is necessaryat any time where the model accuracy data and/or the model efficiencydata compares unfavorably to an indicated model accuracy thresholdand/or indicated model efficiency threshold. The remediation data 650can also include recommissioning instruction data 653, identifyingrequired criteria for recommissioning a medical scan analysis functionand/or updating a medical scan analysis function. The remediation data650 can also include remediation history, indicating one or moreinstances that the medical scan analysis function was taken out ofcommission and/or was recommissioned.

FIGS. 6A and 6B present an embodiment of a medical scan diagnosingsystem 108.

The medical scan diagnosing system 108 can generate inference data 1110for medical scans by utilizing a set of medical scan inference functions1105, stored and run locally, stored and run by another subsystem 101,and/or stored in the medical scan analysis function database 346, wherethe function and/or parameters of the function can be retrieved from thedatabase by the medical scan diagnosing system. For example, the set ofmedical scan inference function 1105 can include some or all medicalscan analysis functions described herein or other functions thatgenerate inference data 1110 based on some or all data corresponding toa medical scan such as some or all data of a medical scan entry 352.Each medical scan inference function 1105 in the set can correspond to ascan category 1120, and can be trained on a set of medical scans thatcompare favorably to the scan category 1120. For example, each inferencefunction can be trained on a set of medical scans of the one or moresame scan classifier data 420, such as the same and/or similar scantypes, same and/or similar anatomical regions locations, same and/orsimilar machine models, same and/or similar machine calibration, sameand/or similar contrasting agent used, same and/or similar originatingentity, same and/or similar geographical region, and/or otherclassifiers. Thus, the scan categories 1120 can correspond to one ormore of a scan type, scan anatomical region data, hospital or otheroriginating entity data, machine model data, machine calibration data,contrast agent data, geographic region data, and/or other scanclassifying data 420. For example, a first medical scan inferencefunction can be directed to characterizing knee x-rays, and a secondmedical scan inference function can be directed to chest CT scans. Asanother example, a first medical scan inference function can be directedto characterizing CT scans from a first hospital, and a second medicalscan image analysis function can be directed to characterizing CT scansfrom a second hospital.

Training on these categorized sets separately can ensure each medicalscan inference function 1105 is calibrated according to its scancategory 1120, for example, allowing different inference functions to becalibrated on type specific, anatomical region specific, hospitalspecific, machine model specific, and/or region-specific tendenciesand/or discrepancies. Some or all of the medical scan inferencefunctions 1105 can be trained by the medical scan image analysis systemand/or the medical scan natural language processing system, and/or somemedical scan inference functions 1105 can utilize both image analysisand natural language analysis techniques to generate inference data1110. For example, some or all of the inference functions can utilizeimage analysis of the medical scan image data 410 and/or naturallanguage data extracted from abnormality annotation data 442 and/orreport data 449 as input, and generate diagnosis data 440 such asmedical codes 447 as output. Each medical scan inference function canutilize the same or different learning models to train on the same ordifferent features of the medical scan data, with the same or differentmodel parameters, for example indicated in the model type data 622 andmodel parameter data 623. Model type and/or parameters can be selectedfor a particular medical scan inference function based on particularcharacteristics of the one or more corresponding scan categories 1120,and some or all of the indicated in the model type data 622 and modelparameter data 623 can be selected automatically by a subsystem duringthe training process based on the particular learned and/or otherwisedetermined characteristics of the one or more corresponding scancategories 1120.

As shown in FIG. 6A, the medical scan diagnosing system 108 canautomatically select a medical scan for processing in response toreceiving it from a medical entity via the network. Alternatively, themedical scan diagnosing system 108 can automatically retrieve a medicalscan from the medical scan database that is selected based on a requestreceived from a user for a particular scan and/or based on a queue ofscans automatically ordered by the medical scan diagnosing system 108 oranother subsystem based on scan priority data 427.

Once a medical scan to be processed is determined, the medical scandiagnosing system 108 can automatically select an inference function1105 based on a determined scan category 1120 of the selected medicalscan and based on corresponding inference function scan categories. Thescan category 1120 of a scan can be determined based one some or all ofthe scan classifier data 420 and/or based on other metadata associatedwith the scan. This can include determining which one of the pluralityof medical scan inference functions 1105 matches or otherwise comparesfavorably to the scan category 1120, for example, by comparing the scancategory 1120 to the input scan category of the function classifier data610.

Alternatively or in addition, the medical scan diagnosing system 108 canautomatically determine which medical scan inference function 1105 isutilized based on an output preference that corresponding to a desiredtype of inference data 1110 that is outputted by an inference function1105. The output preference designated by a user of the medical scandiagnosing system 108 and/or based on the function of a subsystem 101utilizing the medical scan diagnosing system 108. For example, the setof inference functions 1105 can include inference functions that areutilized to indicate whether or not a medical scan is normal, toautomatically identify at least one abnormality in the scan, toautomatically characterize the at least one abnormality in the scan, toassign one or more medical codes to the scan, to generate naturallanguage text data and/or a formatted report for the scan, and/or toautomatically generate other diagnosis data such as some or all ofdiagnosis data 440 based on the medical scan. Alternatively or inaddition, some inference functions can also be utilized to automaticallygenerate confidence score data 460, display parameter data 470, and/orsimilar scan data 480. The medical scan diagnosing system 108 cancompare the output preference to the output type data 612 of the medicalscan inference function 1105 to determine the selected inferencefunction 1105. For example, this can be used to decide between a firstmedical scan inference function that automatically generates medicalcodes and a second medical scan inference function that automaticallygenerates natural language text for medical reports based on the desiredtype of inference data 1110.

Prior to performing the selected medical scan inference function 1105,the medical scan diagnosing system 108 can automatically perform aninput quality assurance function 1106 to ensure the scan classifier data420 or other metadata of the medical scan accurately classifies themedical scan such that the appropriate medical scan inference function1105 of the appropriate scan category 1120 is selected. The inputquality assurance function can be trained on, for example, medical scanimage data 410 of plurality of previous medical scans with verified scancategories. Thus, the input quality assurance function 1106 can takemedical scan image data 410 as input and can generate an inferred scancategory as output. The inferred scan category can be compared to thescan category 1120 of the scan, and the input quality assurance function1106 can determine whether or not the scan category 1120 is appropriateby determining whether the scan category 1120 compares favorably to theautomatically generated inferred scan category. The input qualityassurance function 1106 can also be utilized to reassign the generatedinferred scan category to the scan category 1120 when the scan category1120 compares favorably to the automatically generated inferred scancategory. The input quality assurance function 1106 can also be utilizedto assign the generated inferred scan category to the scan category 1120for incoming medical scans that do not include any classifying data,and/or to add classifiers in scan classifier data 420 to medical scansmissing one or more classifiers.

In various embodiments, upon utilizing the input quality assurancefunction 1106 to determine that the scan category 1120 determined by ascan classifier data 420 or other metadata is inaccurate, the medicalscan diagnosing system 108 can transmit an alert and/or an automaticallygenerated inferred scan category to the medical entity indicating thatthe scan is incorrectly classified in the scan classifier data 420 orother metadata. In some embodiments, the medical scan diagnosing system108 can automatically update performance score data corresponding to theoriginating entity of the scan indicated in originating entity data 423,or another user or entity responsible for classifying the scan, forexample, where a lower performance score is generated in response todetermining that the scan was incorrectly classified and/or where ahigher performance score is generated in response to determining thatthe scan was correctly classified.

In some embodiments, the medical scan diagnosing system 108 can transmitthe medical scan and/or the automatically generated inferred scancategory to a selected user. The user can be presented the medical scanimage data 410 and/or other data of the medical scan via the interactiveinterface 275, for example, displayed in conjunction with the medicalscan assisted review system 102. The interface can prompt the user toindicate the appropriate scan category 1120 and/or prompt the user toconfirm and/or edit the inferred scan category, also presented to theuser. For example, scan review data can be automatically generated toreflect the user generated and/or verified scan category 1120, This userindicated scan category 1120 can be utilized to select to the medicalscan inference function 1105 and/or to update the scan classifier data420 or other metadata accordingly. In some embodiments, for example,where the scan review data indicates that the selected user disagreeswith the automatically generated inferred scan category created by theinput quality assurance function 1106, the medical scan diagnosingsystem 108 can automatically update performance score data 630 of theinput quality assurance function 1106 by generating a low performancescore and/or determine to enter the remediation step 1140 for the inputquality assurance function 1106.

The medical scan diagnosing system 108 can also automatically perform anoutput quality assurance step after a medical scan inference function1105 has been performed on a medical scan to produce the inference data1110, as illustrated in the embodiment presented in FIG. 6B. The outputquality assurance step can be utilized to ensure that the selectedmedical scan inference function 1105 generated appropriate inferencedata 1110 based on expert feedback. The inference data 1110 generated byperforming the selected medical scan inference function 1105 can be sentto a client device 120 of a selected expert user, such as an expert userin the user database selected based on categorized performance dataand/or qualification data that corresponds to the scan category 1120and/or the inference itself, for example, by selecting an expert userbest suited to review an identified abnormality classifier category 444and/or abnormality pattern category 446 in the inference data 1110 basedon categorized performance data and/or qualification data of acorresponding user entry. The selected user can also correspond to amedical professional or other user employed at the originating entityand/or corresponding to the originating medical professional, indicatedin the originating entity data 423.

FIG. 6B illustrates an embodiment of the medical scan diagnosing system108 in conjunction with performing a remediation step 1140. The medicalscan diagnosing system 108 can monitor the performance of the set ofmedical scan inference functions 1105, for example, based on evaluatinginference accuracy data outputted by an inference data evaluationfunction and/or based monitoring on the performance score data 630 inthe medical scan analysis function database, and can determine whetheror not if the corresponding medical scan inference function 1105 isperforming properly. This can include, for example, determining if aremediation step 1140 is necessary for a medical scan inference function1105, for example, by comparing the performance score data 630 and/orinference accuracy data to remediation criteria data 652. Determining ifa remediation step 1140 is necessary can also be based on receiving anindication from the expert user or another user that remediation isnecessary for one or more identified medical scan inference functions1105 and/or for all of the medical scan inference functions 1105.

In various embodiments, a remediation evaluation function is utilized todetermine if a remediation step 1140 is necessary for medical scaninference function 1105. The remediation evaluation function can includedetermining that remediation is necessary when recent accuracy dataand/or efficiency data of a particular medical scan inference function1105 is below the normal performance level of the particular inferencefunction. The remediation evaluation function can include determiningthat remediation is necessary when recent or overall accuracy dataand/or efficiency data of a particular medical scan inference function1105 is below a recent or overall average for all or similar medicalscan inference functions 1105. The remediation evaluation function caninclude determining that remediation is necessary only after a thresholdnumber of incorrect diagnoses are made. In various embodiments, multiplethreshold number of incorrect diagnoses correspond to differentdiagnoses categories. For example, the threshold number of incorrectdiagnoses for remediation can be higher for false negative diagnosesthan false positive diagnoses. Similarly, categories corresponding todifferent diagnosis severities and/or rarities can have differentthresholds, for example where a threshold number of more severe and/ormore rare diagnoses that were inaccurate to necessitate remediation islower than a threshold number of less severe and/or less rare diagnosesthat were inaccurate.

The remediation step 1140 can include automatically updating anidentified medical inference function 1105. This can includeautomatically retraining identified medical inference function 1105 onthe same training set or on a new training set that includes new data,data with higher corresponding confidence scores, or data selected basedon new training set criteria. The identified medical inference function1105 can also be updated and/or changed based on the review datareceived from the client device. For example, the medical scan andexpert feedback data can be added to the training set of the medicalscan inference function 1105, and the medical scan inference function1105 can be retrained on the updated training set. Alternatively or inaddition, the expert user can identify additional parameters and/orrules in the expert feedback data based on the errors made by theinference function in generating the inference data 1110 for the medicalscan, and these parameters and/or rules can be applied to update themedical scan inference function, for example, by updating the model typedata 622 and/or model parameter data 623.

The remediation step 1140 can also include determining to split a scancategory 1120 into two or more subcategories. Thus, two or more newmedical scan inference functions 1105 can be created, where each newmedical scan inference functions 1105 is trained on a correspondingtraining set that is a subset of the original training set and/orincludes new medical scan data corresponding to the subcategory. Thiscan allow medical scan inference functions 1105 to become morespecialized and/or allow functions to utilize characteristics and/ordiscrepancies specific to the subcategory when generating inference data1110. Similarly, a new scan category 1120 that was not previouslyrepresented by any of the medical scan inference functions 1105 can beadded in the remediation step, and a new medical scan inferencefunctions 1105 can be trained on a new set of medical scan data thatcorresponds to the new scan category 1120. Splitting a scan categoryand/or adding a scan category can be determined automatically by themedical scan diagnosing system 108 when performing the remediation step1140, for example, based on performance score data 630. This can also bedetermined based on receiving instructions to split a category and/oradd a new scan category from the expert user or other user of thesystem.

After a medical scan inference function 1105 is updated or created forthe first time, the remediation step 1140 can further undergo acommissioning test, which can include rigorous testing of the medicalscan inference function 1105 on a testing set, for example, based on thetraining parameters 620. For example, the commissioning test can bepassed when the medical scan inference function 1105 generates athreshold number of correct inference data 1110 and/or the test can bepassed if an overall or average discrepancy level between the inferencedata and the test data is below a set error threshold. The commissioningtest can also evaluate efficiency, where the medical scan inferencefunction 1105 only passes the commissioning test if it performs at orexceeds a threshold efficiency level. If the medical scan inferencefunction 1105 fails the commissioning test, the model type and/or modelparameters can be modified automatically or based on user input, and themedical scan inference function can be retested, continuing this processuntil the medical scan inference function 1105 passes the commissioningtest.

The remediation step 1140 can include decommissioning the medical scaninference function 1105, for example, while the medical scan inferencefunction is being retrained and/or is undergoing the commissioning test.Incoming scans to the medical scan diagnosing system 108 with a scancategory 1120 corresponding to a decommissioned medical scan inferencefunction 1105 can be sent directly to review by one or more users, forexample, in conjunction with the medical scan annotator system 106.These user-reviewed medical scans and corresponding annotations can beincluded in an updated training set used to train the decommissionedmedical scan inference function 1105 as part of the remediation step1140. In some embodiments, previous versions of the plurality of medicalscan image analysis functions can be stored in memory of the medicalscan diagnosing system and/or can be determined based on the versiondata 640 of a medical scan inference function 1105. A previous versionof a medical scan inference function 1105, such as most recent versionor version with the highest performance score, can be utilized duringthe remediation step 1140 as an alternative to sending all medical scansto user review.

A medical scan inference function can also undergo the remediation step1140 automatically in response to a hardware and/or software update onprocessing, memory, and/or other computing devices where the medicalscan inference function 1105 is stored and/or performed. Differentmedical scan inference functions 1105 can be containerized on their owndevices by utilizing a micro-service architecture, so hardware and/orsoftware updates may only necessitate that one of the medical scaninference functions 1105 undergo the remediation step 1140 while theothers remain unaffected. A medical scan inference function 1105 canalso undergo the remediation step 1140 automatically in response tonormal system boot-up, and/or periodically in fixed intervals. Forexample, in response to a scheduled or automatically detected hardwareand/or software update, change, or issue, one or more medical scaninference functions 1105 affected by this hardware or software can betaken out of commission until they each pass the commissioning test.Such criteria can be indicated in the remediation criteria data 652.

The medical scan diagnosing system 108 can automatically manage usagedata, subscription data, and/or billing data for the plurality of userscorresponding to user usage of the system, for example, by utilizing,generating, and/or updating some or all of the subscription data of theuser database. Users can pay for subscriptions to the system, which caninclude different subscription levels that can correspond to differentcosts. For example, a hospital can pay a monthly cost to automaticallydiagnose up to 100 medical scans per month. The hospital can choose toupgrade their subscription or pay per-scan costs for automaticdiagnosing of additional scans received after the quota is reachedand/or the medical scan diagnosing system 108 can automatically sendmedical scans received after the quota is reached to an expert userassociated with the hospital. In various embodiments incentive programscan be used by the medical scan diagnosing system to encourage expertsto review medical scans from different medical entities. For example, anexpert can receive credit to their account and/or subscription upgradesfor every medical scan reviewed, or after a threshold number of medicalscans are reviewed. The incentive programs can include interactions by auser with other subsystems, for example, based on contributions made tomedical scan entries via interaction with other subsystems.

FIG. 7A presents an embodiment of a medical scan image analysis system112. A training set of medical scans used to train one more medical scanimage analysis functions can be received from one or more client devicesvia the network and/or can be retrieved from the medical scan database342, for example, based on training set data 621 corresponding tomedical scan image analysis functions. Training set criteria, forexample, identified in training parameters 620 of the medical scan imageanalysis function, can be utilized to automatically identify and selectmedical scans to be included in the training set from a plurality ofavailable medical scans. The training set criteria can be automaticallygenerated based on, for example, previously learned criteria, and/ortraining set criteria can be received via the network, for example, froman administrator of the medical scan image analysis system. The trainingset criteria can include a minimum training set size. The training setcriteria can include data integrity requirements for medical scans inthe training set such as requiring that the medical scan is assigned atruth flag 461, requiring that performance score data for a hospitaland/or medical professional associated with the medical scan comparesfavorably to a performance score threshold, requiring that the medicalscan has been reviewed by at least a threshold number of medicalprofessionals, requiring that the medical scan and/or a diagnosiscorresponding to a patient file of the medical scan is older than athreshold elapsed time period, or based on other criteria intended toinsure that the medical scans and associated data in the training set isreliable enough to be considered “truth” data. The training set criteriacan include longitudinal requirements such the number of requiredsubsequent medical scans for the patient, multiple required types ofadditional scans for the patient, and/or other patient filerequirements.

The training set criteria can include quota and/or proportionrequirements for one or more medical scan classification data. Forexample, the training set criteria can include meeting quota and/orproportion requirements for one or more scan types and/or human bodylocation of scans, meeting quota or proportion requirements for a numberof normal medical scans and a number of medicals scans with identifiedabnormalities, meeting quota and/or proportion requirements for a numberof medical scans with abnormalities in certain locations and/or a numberof medical scans with abnormalities that meet certain size, type, orother characteristics, meeting quota and/or proportion data for a numberof medical scans with certain diagnosis or certain corresponding medicalcodes, and/or meeting other identified quota and/or proportion datarelating to metadata, patient data, or other data associated with themedical scans.

In some embodiments, multiple training sets are created to generatecorresponding medical scan image analysis functions, for example,corresponding to some or all of the set of medical scan inferencefunctions 1105. Some or all training sets can be categorized based onsome or all of the scan classifier data 420 as described in conjunctionwith the medical scan diagnosing system 108, where medical scans areincluded in a training set based on their scan classifier data 420matching the scan category of the training set. In some embodiments, theinput quality assurance function 1106 or another input check step can beperformed on medical scans selected for each training set to confirmthat their corresponding scan classifier data 420 is correct. In someembodiments, the input quality assurance function can correspond to itsown medical scan image analysis function, trained by the medical scanimage analysis system, where the input quality assurance functionutilizes high level computer vision technology to determine a scancategory 1120 and/or to confirm the scan classifier data 420 alreadyassigned to the medical scan.

In some embodiments, the training set will be used to create a singleneural network model, or other model corresponding to model type data622 and/or model parameter data 623 of the medical scan image analysisfunction that can be trained on some or all of the medical scanclassification data described above and/or other metadata, patient data,or other data associated with the medical scans. In other embodiments, aplurality of training sets will be created to generate a plurality ofcorresponding neural network models, where the multiple training setsare divided based on some or all of the medical scan classification datadescribed above and/or other metadata, patient data, or other dataassociated with the medical scans. Each of the plurality of neuralnetwork models can be generated based on the same or different learningalgorithm that utilizes the same or different features of the medicalscans in the corresponding one of the plurality of training sets. Themedical scan classifications selected to segregate the medical scansinto multiple training sets can be received via the network, for examplebased on input to an administrator client device from an administrator.The medical scan classifications selected to segregate the medical scanscan be automatically determined by the medical scan image analysissystem, for example, where an unsupervised clustering algorithm isapplied to the original training set to determine appropriate medicalscan classifications based on the output of the unsupervised clusteringalgorithm.

In embodiments where the medical scan image analysis system is used inconjunction with the medical scan diagnosing system, each of the medicalscan image analysis functions associated with each neural network modelcan correspond to one of the plurality of neural network modelsgenerated by the medical scan image analysis system. For example, eachof the plurality of neural network models can be trained on a trainingset classified on scan type, scan human body location, hospital or otheroriginating entity data, machine model data, machine calibration data,contrast agent data, geographic region data, and/or other scanclassifying data as discussed in conjunction with the medical scandiagnosing system. In embodiments where the training set classifiers arelearned, the medical scan diagnosing system can determine which of themedical scan image analysis functions should be applied based on thelearned classifying criteria used to segregate the original trainingset.

A computer vision-based learning algorithm used to create each neuralnetwork model can include selecting a three-dimensional subregion 1310for each medical scan in the training set. This three-dimensionalsubregion 1310 can correspond to a region that is “sampled” from theentire scan that may represent a small fraction of the entire scan.Recall that a medical scan can include a plurality of orderedcross-sectional image slices. Selecting a three-dimensional subregion1310 can be accomplished by selecting a proper image slice subset 1320of the plurality of cross-sectional image slices from each of theplurality of medical scans, and by further selecting a two-dimensionalsubregion 1330 from each of the selected subset of cross-sectional imageslices of the each of the medical scans. In some embodiments, theselected image slices can include one or more non-consecutive imageslices and thus a plurality of disconnected three-dimensional subregionswill be created. In other embodiments, the selected proper subset of theplurality of image slices correspond to a set of consecutive imageslices, as to ensure that a single, connected three-dimensionalsubregion is selected. In some embodiments, entire scans of the trainingset are used to train the neural network model. In such embodiment, asused herein, the three-dimensional subregion 1310 can refer to all ofthe medical scan image data 410 of a medical scan.

In some embodiments, a density windowing step can be applied to the fullscan or the selected three-dimensional subregion. The density windowingstep can include utilizing a selected upper density value cut off and/ora selected lower density value cut off, and masking pixels with highervalues than the upper density value cut off and/or masking pixels withlower values than the lower density value cut off. The upper densityvalue cut off and/or a selected lower density value cut off can bedetermined based on based on the range and/or distribution of densityvalues included in the region that includes the abnormality, and/orbased on the range and/or distribution of density values associated withthe abnormality itself, based on user input to a subsystem, based ondisplay parameter data associated with the medical scan or associatedwith medical scans of the same type, and/or can be learned in thetraining step. In some embodiments, a non-linear density windowingfunction can be applied to alter the pixel density values, for example,to stretch or compress contrast. In some embodiments, this densitywindowing step can be performed as a data augmenting step, to createadditional training data for a medical scan in accordance with differentdensity windows.

Having determined the subregion training set 1315 of three-dimensionalsubregions 1310 corresponding to the set of full medical scans in thetraining set, the medical scan image analysis system can complete atraining step 1352 by performing a learning algorithm on the pluralityof three-dimensional subregions to generate model parameter data 1355 ofa corresponding learning model. The learning model can include one ormore of a neural network, an artificial neural network, a convolutionalneural network, a Bayesian model, a support vector machine model, acluster analysis model, or other supervised or unsupervised learningmodel. The model parameter data 1355 can generated by performing thelearning algorithm 1350, and the model parameter data 1355 can beutilized to determine the corresponding medical scan image analysisfunctions. For example, some or all of the model parameter data 1355 canbe mapped to the medical scan analysis function in the model parameterdata 623 or can otherwise define the medical scan analysis function.

The training step 1352 can include creating feature vectors for eachthree-dimensional subregion of the training set for use by the learningalgorithm 1350 to generate the model parameter data 1355. The featurevectors can include the pixel data of the three-dimensional subregionssuch as density values and/or grayscale values of each pixel based on adetermined density window. The feature vectors can also include otherfeatures as additional input features or desired output features, suchas known abnormality data such as location and/or classification data,patient history data such as risk factor data or previous medical scans,diagnosis data, responsible medical entity data, scan machinery model orcalibration data, contrast agent data, medical code data, annotationdata that can include raw or processed natural language text data, scantype and/or anatomical region data, or other data associated with theimage, such as some or all data of a medical scan entry 352. Featurescan be selected based on administrator instructions received via thenetwork and/or can be determined based on determining a feature set thatreduces error in classifying error, for example, by performing across-validation step on multiple models created using different featuresets. The feature vector can be split into an input feature vector andoutput feature vector. The input feature vector can include data thatwill be available in subsequent medical scan input, which can includefor example, the three-dimensional subregion pixel data and/or patienthistory data. The output feature vector can include data that will beinferred in in subsequent medical scan input and can include singleoutput value, such as a binary value indicating whether or not themedical scan includes an abnormality or a value corresponding to one ofa plurality of medical codes corresponding to the image. The outputfeature vector can also include multiple values which can includeabnormality location and/or classification data, diagnosis data, orother output. The output feature vector can also include a determinedupper density value cut off and/or lower density value cut off, forexample, characterizing which pixel values were relevant to detectingand/or classifying an abnormality. Features included in the outputfeature vector can be selected to include features that are known in thetraining set, but may not be known in subsequent medical scans such astriaged scans to be diagnosed by the medical scan diagnosing system,and/or scans to be labeled by the medical scan report labeling system.The set of features in the input feature vector and output featurevector, as well as the importance of different features where eachfeature is assigned a corresponding weight, can also be designated inthe model parameter data 1355.

Consider a medical scan image analysis function that utilizes a neuralnetwork. The neural network can include a plurality of layers, whereeach layer includes a plurality of neural nodes. Each node in one layercan have a connection to some or all nodes in the next layer, where eachconnection is defined by a weight value. Thus, the model parameter data1355 can include a weight vector that includes weight values for everyconnection in the network. Alternatively or in addition, the modelparameter data 1355 can include any vector or set of parametersassociated with the neural network model, which can include an upperdensity value cut off and/or lower density value cut off used to masksome of the pixel data of an incoming image, kernel values, filterparameters, bias parameters, and/or parameters characterizing one ormore of a plurality of convolution functions of the neural networkmodel. The medical scan image analysis function can be utilized toproduce the output vector as a function of the input feature vector andthe model parameter data 1355 that characterizes the neural networkmodel. In particular, the medical scan image analysis function caninclude performing a forward propagation step plurality of neuralnetwork layers to produce an inferred output vector based on the weightvector or other model parameter data 1355. Thus, the learning algorithm1350 utilized in conjunction with a neural network model can includedetermining the model parameter data 1355 corresponding to the neuralnetwork model, for example, by populating the weight vector with optimalweights that best reduce output error.

In particular, determining the model parameter data 1355 can includeutilizing a backpropagation strategy. The forward propagation algorithmcan be performed on at least one input feature vector corresponding toat least one medical scan in the training set to propagate the at leastone input feature vector through the plurality of neural network layersbased on initial and/or default model parameter data 1355, such as aninitial weight vector of initial weight values set by an administratoror chosen at random. The at least one output vector generated byperforming the forward propagation algorithm on the at least one inputfeature vector can be compared to the corresponding at least one knownoutput feature vector to determine an output error. Determining theoutput error can include, for example, computing a vector distance suchas the Euclidian distance, or squared Euclidian distance, between theproduced output vector and the known output vector, and/or determiningan average output error such as an average Euclidian distance or squaredEuclidian distance if multiple input feature vectors were employed.Next, gradient descent can be performed to determine an updated weightvector based on the output error or average output error. This gradientdescent step can include computing partial derivatives for the errorwith respect to each weight, or other parameter in the model parameterdata 1355, at each layer starting with the output layer. Chain rule canbe utilized to iteratively compute the gradient with respect to eachweight or parameter at each previous layer until all weight's gradientsare computed. Next updated weights, or other parameters in the modelparameter data 1355, are generated by updating each weight based on itscorresponding calculated gradient. This process can be repeated on atleast one input feature vector, which can include the same or differentat least one feature vector used in the previous iteration, based on theupdated weight vector and/or other updated parameters in the modelparameter data 1355 to create a new updated weight vector and/or othernew updated parameters in the model parameter data 1355. This processcan continue to repeat until the output error converges, the outputerror is within a certain error threshold, or another criterion isreached to determine the most recently updated weight vector and/orother model parameter data 1355 is optimal or otherwise determined forselection.

Having determined the medical scan neural network and its final othermodel parameter data 1355, an inference step 1354 can be performed onnew medical scans to produce inference data 1370, such as inferredoutput vectors, as shown in FIG. 7B. The inference step can includeperforming the forward propagation algorithm to propagate an inputfeature vector through a plurality of neural network layers based on thefinal model parameter data 1355, such as the weight values of the finalweight vector, to produce the inference data. This inference step 1354can correspond to performing the medical scan image analysis function,as defined by the final model parameter data 1355, on new medical scansto generate the inference data 1370, for example, in conjunction withthe medical scan diagnosing system 108 to generate inferred diagnosisdata or other selected output data for triaged medical scans based onits corresponding the input feature vector.

The inference step 1354 can include applying the density windowing stepto new medical scans. Density window cut off values and/or a non-lineardensity windowing function that are learned can be automatically appliedwhen performing the inference step. For example, if the training step1352 was used to determine optimal upper density value cut off and/orlower density value cut off values to designate an optimal densitywindow, the inference step 1354 can include masking pixels of incomingscans that fall outside of this determined density window beforeapplying the forward propagation algorithm. As another example, iflearned parameters of one or more convolutional functions correspond tothe optimal upper density value cut off and/or lower density value cutoff values, the density windowing step is inherently applied when theforward propagation algorithm is performed on the new medical scans.

In some embodiments where a medical scan analysis function is defined bymodel parameter data 1355 corresponding to a neutral network model, theneural network model can be a fully convolutional neural network. Insuch embodiments, only convolution functions are performed to propagatethe input feature vector through the layers of the neural network in theforward propagation algorithm. This enables the medical scan imageanalysis functions to process input feature vectors of any size. Forexample, as discussed herein, the pixel data corresponding to thethree-dimensional subregions is utilized input to the forwardpropagation algorithm when the training step 1352 is employed topopulate the weight vector and/or other model parameter data 1355.However, when performing the forward propagation algorithm in theinference step 1354, the pixel data of full medical scans can beutilized as input, allowing the entire scan to be processed to detectand/or classify abnormalities, or otherwise generate the inference data1370. This may be a preferred embodiment over other embodiments wherenew scans must also be sampled by selecting a three-dimensionalsubregions and/or other embodiments where the inference step requires“piecing together” inference data 1370 corresponding to multiplethree-dimensional subregions processed separately.

The inferred output vector of the inference data 1370 can include aplurality of abnormality probabilities mapped to a pixel location ofeach of a plurality of cross-sectional image slices of the new medicalscan. For example, the inferred output vector can indicate a set ofprobability matrices 1371, where each matrix in the set corresponds toone of the plurality of image slices of the medical scan, where eachmatrix is a size corresponding to the number of pixels in each imageslice, where each cell of each matrix corresponds to a pixel of thecorresponding image slice, whose value is the abnormality probability ofthe corresponding pixel.

A detection step 1372 can include determining if an abnormality ispresent in the medical scan based on the plurality of abnormalityprobabilities. Determining if an abnormality is present can include, forexample, determining that a cluster of pixels in the same region of themedical scan correspond to high abnormality probabilities, for example,where a threshold proportion of abnormality probabilities must meet orexceed a threshold abnormality probability, where an average abnormalityprobability of pixels in the region must meet or exceed a thresholdabnormality probability, where the region that includes the cluster ofpixels must be at least a certain size, etc. Determining if anabnormality is present can also include calculating a confidence scorebased on the abnormality probabilities and/or other data correspondingto the medical scan such as patient history data. The location of thedetected abnormality can be determined in the detection step 1372 basedon the location of the pixels with the high abnormality probabilities.The detection step can further include determining an abnormality region1373, such as a two-dimensional subregion on one or more image slicesthat includes some or all of the abnormality. The abnormality region1373 determined in the detection step 1372 can be mapped to the medicalscan to populate some or all of the abnormality location data 443 foruse by one or more other subsystems 101 and/or client devices 120.Furthermore, determining whether or not an abnormality exists in thedetection step 1372 can be used to populate some or all of the diagnosisdata 440 of the medical scan, for example, to indicate that the scan isnormal or contains an abnormality in the diagnosis data 440.

An abnormality classification step 1374 can be performed on a medicalscan in response to determining an abnormality is present.Classification data 1375 corresponding to one or more classificationcategories such as abnormality size, volume, pre-post contract, doublingtime, calcification, components, smoothness, texture, diagnosis data,one or more medical codes, a malignancy rating such as a Lung-RADSscore, or other classifying data as described herein can be determinedbased on the detected abnormality. The classification data 1375generated by the abnormality classification step 1374 can be mapped tothe medical scan to populate some or all of the abnormalityclassification data 445 of the corresponding abnormality classifiercategories 444 and/or abnormality pattern categories 446 and/or todetermine one or more medical codes 447 of the medical scan. Theabnormality classification step 1374 can include performing anabnormality classification function on the full medical scan, or theabnormality region 1373 determined in the detection step 1372. Theabnormality classification function can be based on another modeltrained on abnormality data such as a support vector machine model,another neural network model, or any supervised classification modeltrained on medical scans, or portions of medical scans, that includeknown abnormality classifying data to generate inference data for someor all of the classification categories. For example, the abnormalityclassification function can include another medical scan analysisfunction. Classification data 1375 in each of a plurality ofclassification categories can also be assigned their own calculatedconfidence score, which can also be generated by utilizing theabnormality classification function. Output to the abnormalityclassification function can also include at least one identified similarmedical scan and/or at least one identified similar cropped image, forexample, based on the training data. The abnormality classification stepcan also be included in the inference step 1354, where the inferredoutput vector or other inference data 1370 of the medical scan imageanalysis function includes the classification data 1375.

The abnormality classification function can be trained on full medicalscans and/or one or more cropped or full selected image slices frommedical scans that contain an abnormality. For example, the abnormalityclassification function can be trained on a set of two-dimensionalcropped slices that include abnormalities. The selected image slicesand/or the cropped region in each selected image slice for each scan inthe training set can be automatically selected based upon the knownlocation of the abnormality. Input to the abnormality classificationfunction can include the full medical scan, one or more selected fullimage slices, and/or one or more selected image slices cropped based ona selected region. Thus, the abnormality classification step can includeautomatically selecting one or more image slices that include thedetected abnormality. The slice selection can include selecting thecenter slice in a set of consecutive slices that are determined toinclude the abnormality or selecting a slice that has the largestcross-section of the abnormality, or selecting one or more slices basedon other criteria. The abnormality classification step can also includeautomatically generating one or more cropped two-dimensional imagescorresponding to the one or more of the selected image slices based onan automatically selected region that includes the abnormality.

Input to the abnormality classification function can also include otherdata associated with the medical scan, including patient history, riskfactors, or other metadata. The abnormality classification step can alsoinclude determining some or all of the characteristics based on data ofthe medical scan itself. For example, the abnormality size and volumecan be determined based on a number of pixels determined to be part ofthe detected abnormality. Other classifiers such as abnormality textureand/or smoothness can be determined by performing one or more otherpreprocessing functions on the image specifically designed tocharacterize such features. Such preprocessed characteristics can beincluded in the input to the abnormality classification function to themore difficult task of assigning a medical code or generating otherdiagnosis data. The training data can also be preprocessed to includesuch preprocessed features.

A similar scan identification step 1376 can also be performed on amedical scan with a detected abnormality and/or can be performed on theabnormality region 1373 determined in the detection step 1372. Thesimilar scan identification step 1376 can include generating similarabnormality data 1377, for example, by identifying one or more similarmedical scans or one or more similar cropped two-dimensional images froma database of medical scans and/or database of cropped two-dimensionalimages. Similar medical scans and/or cropped images can include medicalscans or cropped images that are visually similar, medical scans orcropped images that have known abnormalities in a similar location to aninferred abnormality location of the given medical scan, medical scansthat have known abnormalities with similar characteristics to inferredcharacteristics of an abnormality in the given scan, medical scans withsimilar patient history and/or similar risk factors, or some combinationof these factors and/or other known and/or inferred factors. The similarabnormality data 1377 can be mapped to the medical scan to populate someor all of its corresponding similar scan data 480 for use by one or moreother subsystems 101 and/or client devices 120.

The similar scans identification step 1376 can include performing a scansimilarity algorithm, which can include generating a feature vector forthe given medical scan and for medical scans in the set of medicalscans, where the feature vector can be generated based on quantitativeand/or category based visual features, inferred features, abnormalitylocation and/or characteristics such as the predetermined size and/orvolume, patient history and/or risk factor features, or other known orinferred features. A medical scan similarity analysis function can beapplied to the feature vector of the given medical scan and one or morefeature vectors of medical scans in the set. The medical scan similarityanalysis function can include computing a similarity distance such asthe Euclidian distance between the feature vectors, and assigning thesimilarity distance to the corresponding medical scan in the set.Similar medical scans can be identified based on determining one or moremedical scans in the set with a smallest computed similarity distance,based on ranking medical scans in the set based on the computedsimilarity distances and identifying a designated number of top rankedmedical scans, and/or based on determining if a similarity distancebetween the given medical scan and a medical scan in the set is smallerthan a similarity threshold. Similar medical scans can also beidentified based on determining medical scans in a database that mappedto a medical code that matches the medical code of the medical scan, ormapped to other matching classifying data. A set of identified similarmedical scans can also be filtered based on other inputted orautomatically generated criteria, where for example only medical scanswith reliable diagnosis data or rich patient reports, medical scans withcorresponding with longitudinal data in the patient file such asmultiple subsequent scans taken at later dates, medical scans withpatient data that corresponds to risk factors of the given patient, orother identified criteria, where only a subset of scans that comparefavorably to the criteria are selected from the set and/or only ahighest ranked single scan or subset of scans are selected from the set,where the ranking is automatically computed based on the criteria.Filtering the similar scans in this fashion can include calculating, orcan be based on previously calculated, one or more scores as discussedherein. For example, the ranking can be based on a longitudinal qualityscore, such as the longitudinal quality score 434, which can becalculated for an identified medical scan based on a number ofsubsequent and/or previous scans for the patient. Alternatively or inaddition, the ranking can be based on a confidence score associated withdiagnosis data of the scan, such as confidence score data 460, based onperformance score data associated with a user or medical entityassociated with the scan, based on an amount of patient history data ordata in the medical scan entry 352, or other quality factors. Theidentified similar medical scans can be filtered based on ranking thescans based on their quality score and/or based on comparing theirquality score to a quality score threshold. In some embodiments, alongitudinal threshold must be reached, and only scans that comparefavorably to the longitudinal threshold will be selected. For example,only scans with at least three scans on file for the patient and finalbiopsy data will be included.

In some embodiments, the similarity algorithm can be utilized inaddition to or instead of the trained abnormality classificationfunction to determine some or all of the inferred classification data1375 of the medical scan, based on the classification data such asabnormality classification data 445 or other diagnosis data 440 mappedto one or more of the identified similar scans. In other embodiments,the similarity algorithm is merely used to identify similar scans forreview by medical professionals to aid in review, diagnosis, and/orgenerating medical reports for the medical image.

A display parameter step 1378 can be performed based on the detectionand/or classification of the abnormality. The display parameter step caninclude generating display parameter data 1379, which can includeparameters that can be used by an interactive interface to best displayeach abnormality. The same or different display parameters can begenerated for each abnormality. The display parameter data generated inthe display parameter step 1378 can be mapped to the medical scan topopulate some or all of its corresponding display parameter data 470 foruse by one or more other subsystems 101 and/or client devices 120.

Performing the display parameter step 1378 can include selecting one ormore image slices that include the abnormality by determining the one ormore image slices that include the abnormality and/or determining one ormore image slices that has a most optimal two-dimensional view of theabnormality, for example by selecting the center slice in a set ofconsecutive slices that are determined to include the abnormality,selecting a slice that has the largest cross-section of the abnormality,selecting a slice that includes a two-dimensional image of theabnormality that is most similar to a selected most similartwo-dimensional-image, selecting the slice that was used as input to theabnormality classification step and/or similar scan identification step,or based on other criteria. This can also include automatically croppingone or more selected image slices based on an identified region thatincludes the abnormality. This can also select an ideal Hounsfieldwindow that best displays the abnormality. This can also includeselecting other display parameters based on data generated by themedical scan interface evaluating system and based on the medical scan.

FIGS. 8A-8F illustrate embodiments of a medical picture archiveintegration system 2600. The medical picture archive integration system2600 can provide integration support for a medical picture archivesystem 2620, such as a PACS that stores medical scans. The medicalpicture archive integration system 2600 can utilize model parametersreceived from a central server system 2640 via a network 2630 to performan inference function on de-identified medical scans of medical scansreceived from the medical picture archive system 2620. The annotationdata produced by performing the inference function can be transmittedback to the medical picture archive system. Furthermore, the annotationdata and/or de-identified medical scans can be sent to the centralserver system 2640, and the central server system can train on thisinformation to produce new and/or updated model parameters fortransmission back to the medical picture archive integration system 2600for use on subsequently received medical scans.

In various embodiments, medical picture archive integration system 2600includes a de-identification system that includes a first memorydesignated for protected health information (PHI), operable to perform ade-identification function on a DICOM image, received from a medicalpicture archive system, to identify at least one patient identifier andgenerate a de-identified medical scan that does not include the at leastone patient identifier. The medical picture archive integration systemfurther includes a de-identified image storage system that stores thede-identified medical scan in a second memory that is separate from thefirst memory, and an annotating system, operable to utilize modelparameters received from a central server to perform an inferencefunction on the de-identified medical scan, retrieved from the secondmemory to generate annotation data for transmission to the medicalpicture archive system as an annotated DICOM file.

The first memory and the second memory can be implemented by utilizingseparate storage systems: the first memory can be implemented by a firststorage system designated for PHI storage, and the second memory can beimplemented by a second storage system designated for storage ofde-identified data. The first storage system can be protected fromaccess by the annotating system, while the second storage system can beaccessible by the annotating system. The medical picture archiveintegration system 2600 can be operable to perform the de-identificationfunction on data in first storage system to generate de-identified data.The de-identified data can then be stored in the second storage systemfor access by the annotating system. The first and second storagesystems can be physically separate, each utilizing at least one of theirown, separate memory devices. Alternatively, the first and secondstorage systems can be virtually separate, where data is stored inseparate virtual memory locations on the same set of memory devices.Firewalls, virtual machines, and/or other protected containerization canbe utilized to enforce the separation of data in each storage system, toprotect the first storage system from access by the annotating systemand/or from other unauthorized access, and/or to ensure that only dataof the first storage system that has been properly de-identified throughapplication of the de-identification function can be stored in thesecond storage system.

As shown in FIG. 8A, the medical picture archive system 2620 can receiveimage data from a plurality of modality machines 2622, such as CTmachines, Mill machines, x-ray machines, and/or other medical imagingmachines that produce medical scans. The medical picture archive system2620 can store this image data in a DICOM image format and/or can storethe image data in a plurality of medical scan entries 352 as describedin conjunction with some or all of the attributes described inconjunction with FIGS. 4A and 4B. While “DICOM image” will be usedherein to refer to medical scans stored by the medical picture archivesystem 2620, the medical picture archive integration system 2600 canprovide integration support for medical picture archive systems 2620that store medical scans in other formats.

The medical picture archive integration system 2600 can include areceiver 2602 and a transmitter 2604, operable to transmit and receivedata from the medical picture archive system 2620, respectively. Forexample, the receiver 2602 and transmitter 2604 can be configured toreceive and transmit data, respectively, in accordance with a DICOMcommunication protocol and/or another communication protocol recognizedby the medical image archive system 2620. The receiver can receive DICOMimages from the medical picture archive system 2620. The transmitter2604 can send annotated DICOM files to the medical picture archivesystem 2620.

DICOM images received via receiver 2602 can be sent directly to ade-identification system 2608. The de-identification system 2608 can beoperable to perform a de-identification function on the first DICOMimage to identify at least one patient identifier in the DICOM image,and to generate a de-identified medical scan that does not include theidentified at least one patient identifier. As used herein, a patientidentifier can include any patient identifying data in the image data,header, and/or metadata of a medical scan, such as a patient ID numberor other unique patient identifier, an accession number, aservice-object pair (SOP) instance unique identifier (UID) field, scandate and/or time that can be used to determine the identity of thepatient that was scanned at that date and/or time, and/or other privatedata corresponding to the patient, doctor, or hospital. In someembodiments, the de-identified medical scan is still in a DICOM imageformat. For example, a duplicate DICOM image that does not include thepatient identifiers can be generated, and/or the original DICOM imagecan be altered such that the patient identifiers of the new DICOM imageare masked, obfuscated, removed, replaced with a custom fiducial, and/orotherwise anonymized. In other embodiments, the de-identified medicalscan is formatted in accordance with a different image format and/ordifferent data format that does not include the identifying information.In some embodiments, other private information, for example, associatedwith a particular doctor or other medical professional, can beidentified and anonymized as well.

Some patient identifying information can be included in a DICOM headerof the DICOM image, for example, in designated fields for patientidentifiers. These corresponding fields can be anonymized within thecorresponding DICOM header field. Other patient identifying informationcan be included in the image itself, such as in medical scan image data410. For example, the image data can include a patient name or otheridentifier that was handwritten on a hard copy of the image before theimage was digitized. As another example, a hospital administered armbandor other visual patient information in the vicinity of the patient mayhave been captured in the image itself. A computer vision model candetect the presence of these identifiers for anonymization, for example,where a new DICOM image includes a fiducial image that covers theidentifying portion of the original DICOM image. In some embodiments,patient information identified in the DICOM header can be utilized todetect corresponding patient information in the image itself. Forexample, a patient name extracted from the DICOM header beforeanonymization can be used to search for the patient name in the imageand/or to detect a location of the image that includes the patient name.In some embodiments, the de-identification system 2608 is implemented bythe de-identification system discussed in conjunction with FIGS. 10A,10B and 11, and/or utilizes functions and/or operations discussed inconjunction with FIGS. 10A, 10B and 11.

The de-identified medical scan can be stored in de-identified imagestorage system 2610 and the annotating system 2612 can access thede-identified medical scan from the de-identified image storage system2610 for processing. The de-identified storage system can archive aplurality of de-identified DICOM images and/or can serve as temporarystorage for the de-identified medical scan until processing of thede-identified medical scan by the annotating system 2612 is complete.The annotating system 2612 can generate annotation data by performing aninference function on the de-identified medical scan, utilizing themodel parameters received from the central server system 2640. Theannotation data can correspond to some or all of the diagnosis data 440as discussed in conjunction with FIGS. 4A and 4B. In come embodiments,the annotating system 2612 can utilize the model parameters to performinference step 1354, the detection step 1372, the abnormalityclassification step 1374, the similar scan identification step 1376,and/or the display parameter step 1378 of the medical scan imageanalysis system 112, as discussed in conjunction with FIG. 7B, onde-identified medical scans received from the medical picture archivesystem 2620.

In some embodiments, model parameters for a plurality of inferencefunctions can be received from the central server system 2640, forexample, where each inference function corresponds to one of a set ofdifferent scan categories. Each scan category can correspond to a uniquecombination of one or a plurality of scan modalities, one of a pluralityof anatomical regions, and/or other scan classifier data 420. Forexample, a first inference function can be trained on and intended forde-identified medical scans corresponding chest CT scans, and a secondinference function can be trained on and intended for de-identifiedmedical scans corresponding to head MM scans. The annotating system canselect one of the set of inference functions based on determining thescan category of the DICOM image, indicated in the de-identified medicalscan, and selecting the inference function that corresponds to thedetermined scan category.

To ensure that scans received from the medical picture archive system2620 match the set of scan categories for which the annotating system isoperable to perform a corresponding inference function, the transmittercan transmit requests, such as DICOM queries, indicating image typeparameters such as parameters corresponding to scan classifier data 420,for example indicating one or more scan modalities, one or moreanatomical regions, and/or other parameters. For example, the requestcan indicate that all incoming scans that match the set of scancategories corresponding to a set of inference functions the annotatingsystem 2612 for which the annotating system has obtained modelparameters from the central server system 2640 and is operable toperform.

Once the annotation data is generated by performing the selectedinference function, the annotating system 2612 can generate an annotatedDICOM file for transmission to the medical image archive system 2620 forstorage. The annotated DICOM file can include some or all of the fieldsof the diagnosis data 440 and/or abnormality annotation data 442 ofFIGS. 4A and 4B. The annotated DICOM file can include scan overlay data,providing location data of an identified abnormality and/or display datathat can be used in conjunction with the original DICOM image toindicate the abnormality visually in the DICOM image and/or to otherwisevisually present the annotation data, for example, for use with themedical scan assisted review system 102. For example, a DICOMpresentation state file can be generated to indicate the location of anabnormality identified in the de-identified medical scan. The DICOMpresentation state file can include an identifier of the original DICOMimage, for example, in metadata of the DICOM presentation state file, tolink the annotation data to the original DICOM image. In otherembodiments, a full, duplicate DICOM image is generated that includesthe annotation data with an identifier linking this duplicate annotatedDICOM image to the original DICOM image.

The identifier linking the annotated DICOM file to the original DICOMimage can be extracted from the original DICOM file by thede-identification system 2608, thus enabling the medical picture archivesystem 2620 to link the annotated DICOM file to the original DICOM imagein its storage. For example, the de-identified medical scan can includean identifier that links the de-identified medical scan to the originalDICOM file, but does not link the de-identified medical scan to apatient identifier or other private data.

In some embodiments, generating the annotated DICOM file includesaltering one or more fields of the original DICOM header. For example,standardized header formatting function parameters can be received fromthe central server system and can be utilized by the annotating systemto alter the original DICOM header to match a standardized DICOM headerformat. The standardized header formatting function can be trained in asimilar fashion to other medical scan analysis functions discussedherein and/or can be characterized by some or all fields of a medicalscan analysis function entry 356. The annotating system can perform thestandardized header formatting function on a de-identified medical scanto generate a new, standardized DICOM header for the medical scan to besent back to the medical picture archive system 2620 in the annotatedDICOM file and/or to replace the header of the original DICOM file. Thestandardized header formatting function can be run in addition to otherinference functions utilized to generate annotation data. In otherembodiments, the medical picture archive integration system 2600 isimplemented primarily for header standardization for medical scansstored by the medical picture archive system 2620. In such embodiments,only the standardized header formatting function is performed on thede-identified data to generate a modified DICOM header for the originalDICOM image, but the de-identified medical scan is not annotated.

In some embodiments of header standardization, the annotation system canstore a set of acceptable, standardized entries for some or all of theDICOM header fields, and can select one of the set of acceptable,standardized entries in populating one or more fields of the new DICOMheader for the annotated DICOM file. For example, each of the set ofscan categories determined by the annotating system can correspond to astandardized entry of one or more fields of the DICOM header. The newDICOM header can thus be populated based on the determined scancategory.

In some embodiments, each of the set of standardized entries can bemapped to a set of related, non-standardized entries, such as entries ina different order, commonly misspelled entries, or other similar entriesthat do not follow a standardized format. For example, one of the set ofacceptable, standardized entries for a field corresponding to a scancategory can include “Chest CT”, which can be mapped to a set ofsimilar, non-standardized entries which can include “CT chest”,“computerized topography CT”, and/or other entries that are notstandardized. In such embodiments, the annotating system can determinethe original DICOM header is one of the similar non-standardizedentries, and can select the mapped, standardized entry as the entry forthe modified DICOM header. In other embodiments, the image data itselfand/or or other header data can be utilized by the annotation system todetermine a standardized field. For example, an input quality assurancefunction 1106 can be trained by the central server system and sent tothe annotating system to determine one or more appropriate scanclassifier fields, or one or more other DICOM header fields, based onthe image data or other data of the de-identified medical scan. One ormore standardized labels can be assigned to corresponding fields of themodified DICOM header based on the one or more fields determined by theinput quality assurance function.

In some embodiments, the DICOM header is modified based on theannotation data generated in performing the inference function. Inparticular, a DICOM priority header field can be generated and/ormodified automatically based on the severity and/or time-sensitivity ofthe abnormalities detected in performing the inference function. Forexample, a DICOM priority header field can be changed from a lowpriority to a high priority in response to annotation data indicating abrain bleed in the de-identified medical scan of a DICOM imagecorresponding to a head CT scan, and a new DICOM header that includesthe high priority DICOM priority header field can be sent back to themedical picture archive system 2620 to replace or otherwise be mapped tothe original DICOM image of the head CT scan.

In various embodiments, the medical picture archive system 2620 isdisconnected from network 2630, for example, to comply with requirementsregarding Protected Health Information (PHI), such as patientidentifiers and other private patient information included in the DICOMimages and/or otherwise stored by the medical picture archive system2620. The medical picture archive integration system 2600 can enableprocessing of DICOM images while still protecting private patientinformation by first de-identifying DICOM data by utilizingde-identification system 2608. The de-identification system 2608 canutilize designated processors and memory of the medical picture archiveintegration system, for example, designated for PHI. Thede-identification system 2608 can be decoupled from the network 2630 toprevent the DICOM images that still include patient identifiers frombeing accessed via the network 2630. For example, as shown in FIG. 8A,the de-identification system 2608 is not connected to network interface2606. Furthermore, only the de-identification system 2608 has access tothe original DICOM files received from the medical picture archivesystem 2620 via receiver 2602. The de-identified image storage system2610 and annotating system 2612, as they are connected to network 2630via network interface 2606, only store and have access to thede-identified medical scan produced by the de-identification system2608.

This containerization that separates the de-identification system 2608from the de-identified image storage system 2610 and the annotatingsystem 2612 is further illustrated in FIG. 8B, which presents anembodiment of the medical picture archive integration system 2600. Thede-identification system 2608 can include its own designated memory 2654and processing system 2652, connected to receiver 2602 via bus 2659. Forexample, this memory 2654 and processing system 2652 can be designatedfor PHI, and can adhere to requirements for handling PHI. The memory2654 can store executable instructions that, when executed by theprocessing system 2652, enable the de-identification system to performthe de-identification function on DICOM images received via receiver2602 of the de-identification system. The incoming DICOM images can betemporarily stored in memory 2654 for processing, and patientidentifiers detected in performing the de-identification function can betemporarily stored in memory 2654 to undergo anonymization. Interface2655 can transmit the de-identified medical scan to interface 2661 foruse by the de-identified image storage system 2610 and the annotatingsystem 2612. Interface 2655 can be protected from transmitting originalDICOM files and can be designated for transmission of de-identifiedmedical scan only.

Bus 2669 connects interface 2661, as well as transmitter 2604 andnetwork interface 2606, to the de-identified image storage system 2610and the annotating system 2612. The de-identified image storage system2610 and annotating system 2612 can utilize separate processors andmemory, or can utilize shared processors and/or memory. For example, thede-identified image storage system 2610 can serve as temporary memory ofthe annotating system 2612 as de-identified images are received andprocessed to generate annotation data.

As depicted in FIG. 8B, the de-identified image storage system 2610 caninclude memory 2674 that can temporarily store incoming de-identifiedmedical scans as it undergoes processing by the annotating system 2612and/or can archive a plurality of de-identified medical scanscorresponding to a plurality of DICOM images received by the medicalpicture archive integration system 2600. The annotating system 2612 caninclude a memory 2684 that stores executable instructions that, whenexecuted by processing system 2682, cause the annotating system 2612perform a first inference function on de-identified medical scan togenerate annotation data by utilizing the model parameters received viainterface 2606, and to generate an annotated DICOM file based on theannotation data for transmission via transmitter 2604. The modelparameters can be stored in memory 2684, and can include modelparameters for a plurality of inference functions, for example,corresponding to a set of different scan categories.

The medical picture archive integration system can be an onsite system,installed at a first geographic site, such as a hospital or othermedical entity that is affiliated with the medical picture archivesystem 2620. The hospital or other medical entity can further beresponsible for the PHI of the de-identification system, for example,where the memory 2654 and processing system 2652 are owned by,maintained by, and/or otherwise affiliated with the hospital or othermedical entity. The central server system 2640 can be located at asecond, separate geographic site that is not affiliated with thehospital or other medical entity and/or at a separate geographic sitethat is not affiliated with the medical picture archive system 2620. Thecentral server system 2640 can be a server configured to be outside thenetwork firewall and/or out outside the physical security of thehospital or other medical entity or otherwise not covered by theparticular administrative, physical and technical safeguards of thehospital or other medical entity.

FIG. 8C further illustrates how model parameters can be updated overtime to improve existing inference functions and/or to add new inferencefunctions, for example corresponding to new scan categories. Inparticular, the some or all of the de-identified medical scans generatedby the de-identification system 2608 can be transmitted back to thecentral server system, and the central server system 2640 can train onthis data to improve existing models by producing updated modelparameters of an existing inference function and/or to generate newmodels, for example, corresponding to new scan categories, by producingnew model parameters for new inference functions. For example, thecentral server system 2640 can produce updated and/or new modelparameters by performing the training step 1352 of the medical scanimage analysis system 112, as discussed in conjunction with FIG. 7A, ona plurality of de-identified medical scans received from the medicalpicture archive integration system 2600.

The image type parameters can be determined by the central server systemto dictate characteristics of the set of de-identified medical scans tobe received to train and/or retrain the model. For example, the imagetype parameters can correspond to one or more scan categories, canindicate scan classifier data 420, can indicate one or more scanmodalities, one or more anatomical regions, a date range, and/or otherparameters. The image type parameters can be determined by the centralserver system based on training parameters 620 determined for thecorresponding inference function to be trained, and/or based oncharacteristics of a new and/or existing scan category corresponding tothe inference function to be trained. The image type parameters can besent to the medical picture archive integration system 2600, and arequest such as a DICOM query can be sent to the medical picture archivesystem 2620, via transmitter 2604, that indicates the image typeparameters. For example, the processing system 2682 can be utilized togenerate the DICOM query based on the image type parameters receivedfrom the central server system 2640. The medical picture archive systemcan automatically transmit one or more DICOM images to the medicalpicture archive integration system in response to determining that theone or more DICOM images compares favorably to the image typeparameters. The DICOM images received in response can be de-identifiedby the de-identification system 2608. In some embodiments, thede-identified medical scans can be transmitted directly to the centralserver system 2640, for example, without generating annotation data.

The central server system can generate the new and/or updated modelparameters by training on the received set of de-identified medicalscans, and can transmit the new and/or updated model parameters to thede-identified storage system. If the model parameters correspond to anew inference function for a new scan category, the medical picturearchive integration system 2600 can generate a request, such as a DICOMquery, for transmission to the medical picture archive system indicatingthat incoming scans corresponding to image type parameters correspondingto the new scan category be sent to the medical picture archiveintegration system. The annotating system can update the set ofinference functions to include the new inference function, and theannotating system can select the new inference function from the set ofinference functions for subsequently generated de-identified medicalscans by the de-identification system by determining each of thesede-identified medical scans indicate the corresponding DICOM imagecorresponds to the new scan category. The new model parameters can beutilized to perform the new inference function on each of thesede-identified medical scans to generate corresponding annotation data,and an annotated DICOM file corresponding to each of these de-identifiedmedical scans can be generated for transmission to the medical picturearchive system via the transmitter.

In some embodiments, the central server system 2640 receives a pluralityof de-identified medical scans from a plurality of medical picturearchive integration system 2600, for example, each installed at aplurality of different hospitals or other medical entities, via thenetwork 2630. The central server system can generate training sets byintegrating de-identified medical scans from some or all of theplurality of medical picture archive integration systems 2600 to trainone or more inference functions and generate model parameters. Theplurality of medical picture archive integration systems 2600 canutilize the same set of inference functions or different sets ofinference functions. In some embodiments, the set of inference functionsutilized by the each of the plurality of medical picture archive systems2620 are trained on different sets of training data. For example, thedifferent sets of training data can correspond to the set ofde-identified medical scans received from the corresponding medicalpicture archive integration system 2600.

In some embodiments, the medical scan diagnosing system 108 can beutilized to implement the annotating system 2612, where thecorresponding subsystem processing device 235 and subsystem memorydevice 245 of the medical scan diagnosing system 108 are utilized toimplement the processing system 2682 and the memory 2684, respectively.Rather than receiving the medical scans via the network 150 as discussedin conjunction with FIG. 6A, the medical scan diagnosing system 108 canperform a selected medical scan inference function 1105 on an incomingde-identified medical scan generated by the de-identification system2608 and/or retrieved from the de-identified image storage system 2610.Memory 2684 can store the set of medical scan inference functions 1105,each corresponding to a scan category 1120, where the inference functionis selected from the set based on determining the scan category of thede-identified medical scan and selecting the corresponding inferencefunction. The processing system 2682 can perform the selected inferencefunction 1105 to generate the inference data 1110, which can be furtherutilized by the annotating system 2612 to generate the annotated DICOMfile for transmission back to the medical picture archive system 2620.New medical scan inference functions 1105 can be added to the set whencorresponding model parameters are received from the central serversystem. The remediation step 1140 can be performed locally by theannotating system 2612 and/or can be performed by the central serversystem 2640 by utilizing one or more de-identified medical scans andcorresponding annotation data sent to the central server system 2640.Updated model parameters can be generated by the central server system2640 and sent to the medical picture archive integration system 2600 asa result of performing the remediation step 1140.

The central server system 2640 can be implemented by utilizing one ormore of the medical scan subsystems 101, such as the medical scan imageanalysis system 112 and/or the medical scan diagnosing system 108, toproduce model parameters for one or more inference functions. Thecentral server system can store or otherwise communicate with a medicalscan database 342 that includes the de-identified medical scans and/orannotation data received from one or more medical picture archiveintegration systems 2600. Some or all entries of the medical scandatabase 342 can be utilized to as training data to produce modelparameters for one or more inference functions. These entries of themedical scan database 342 can be utilized by other subsystems 101 asdiscussed herein. For example, other subsystems 101 can utilize thecentral server system 2640 to fetch medical scans and/or correspondingannotation data that meet specified criteria. The central server system2640 can query the medical picture archive integration system 2600 basedon this criteria, and can receive de-identified medical scans and/orannotation data in response. This can be sent to the requestingsubsystem 101 directly and/or can be added to the medical scan database342 or another database of the database storage system 140 for access bythe requesting subsystem 101.

Alternatively or in addition, the central server system 2640 can storeor otherwise communicate with a user database 344 storing user profileentries corresponding to each of a plurality of medical entities thateach utilize a corresponding one of a plurality of medical picturearchive integration systems 2600. For example, basic user datacorresponding to the medical entity can be stored as basic user data, anumber of scans or other consumption information indicating usage of oneor more inference functions by corresponding medical picture archiveintegration system can be stored as consumption usage data, and/or anumber of scans or other contribution information indicatingde-identified scans sent to the central server system as training datacan be stored as contribution usage data. The user profile entry canalso include inference function data, for example, with a list of modelparameters or function identifiers, such as medical scan analysisfunction identifiers 357, of inference functions currently utilized bythe corresponding medical picture archive integration system 2600. Theseentries of the user database 344 can be utilized by other subsystems 101as discussed herein.

Alternatively or in addition, the central server system 2640 can storeor otherwise communicate with a medical scan analysis function database346 to store model parameters, training data, or other information forone or more inference functions as medical scan analysis functionentries 356. In some embodiments, model parameter data 623 can indicatethe model parameters and function classifier data 610 can indicate thescan category of inference function entries. In some embodiments, themedical scan analysis function entry 356 can further include usageidentifying information indicating a medical picture archive integrationsystem identifier, medical entity identifier, and/or otherwiseindicating which medical archive integration systems and/or medicalentities have received the corresponding model parameters to utilize theinference function corresponding to the medical scan analysis functionentry 356. These entries of the medical scan analysis function database346 can be utilized by other subsystems 101 as discussed herein.

In some embodiments, the de-identification function is a medical scananalysis function, for example, with a corresponding medical scananalysis function entry 356 in the medical scan analysis functiondatabase 346. In some embodiments, the de-identification function istrained by the central server system 2640. For example, the centralserver system 2640 can send de-identification function parameters to themedical picture archive integration system 2600 for use by thede-identification system 2608. In embodiments with a plurality ofmedical picture archive integration systems 2600, each of the pluralityof medical picture archive integration systems 2600 can utilize the sameor different de-identification functions. In some embodiments, thede-identification function utilized by the each of the plurality ofmedical picture archive integration systems 2600 are trained ondifferent sets of training data. For example, the different sets oftraining data can correspond to each different set of de-identifiedmedical scans received from each corresponding medical picture archiveintegration system 2600.

In some embodiments, as illustrated in FIGS. 8D-8F, the medical picturearchive integration system 2600 can further communicate with a reportdatabase 2625, such as a Radiology Information System (RIS), thatincludes a plurality of medical reports corresponding to the DICOMimages stored by the medical picture archive system 2620.

As shown in FIG. 8D, the medical picture archive integration system 2600can further include a receiver 2603 that receives report data,corresponding to the DICOM image, from report database 2625. The reportdatabase 2625 can be affiliated with the medical picture archive system2620 and can store report data corresponding to DICOM images stored inthe medical picture archive system. The report data of report database2625 can include PHI, and the report database 2625 can thus bedisconnected from network 2630.

The report data can include natural language text, for example,generated by a radiologist that reviewed the corresponding DICOM image.The report data can be used to generate the de-identified medical scan,for example, where the de-identification system 2608 performs a naturallanguage analysis function on the report data to identify patientidentifying text in the report data. The de-identification system 2608can utilize this patient identifying text to detect matching patientidentifiers in the DICOM image to identify the patient identifiers ofthe DICOM image and generate the de-identified medical scan. In someembodiments, the report data can be de-identified by obfuscating,hashing, removing, replacing with a fiducial, or otherwise anonymizingthe identified patient identifying text to generate de-identified reportdata.

The de-identified report data can be utilized by the annotating system2612, for example, in conjunction with the DICOM image, to generate theannotation data. For example, the annotating system 2612 can perform anatural language analysis function on the de-identified natural languagetext of the report data to generate some or all of the annotation data.In some embodiments, the de-identified report data is sent to thecentral server system, for example, to be used as training data forinference functions, for natural language analysis functions, for othermedical scan analysis functions, and/or for use by at least one othersubsystem 101. For example, other subsystems 101 can utilize the centralserver system 2640 to fetch medical reports that correspond toparticular medical scans or otherwise meet specified criteria. Thecentral server system 2640 can query the medical picture archiveintegration system 2600 based on this criteria, and can receivede-identified medical reports in response. This can be sent to therequesting subsystem 101 directly, can be added to the medical scandatabase 342, a de-identified report database, or another database ofthe database storage system 140 for access by the requesting subsystem101.

In some embodiments the medical picture archive integration system 2600can query the report database 2625 for the report data corresponding toa received DICOM image by utilizing a common identifier extracted fromthe DICOM image.

In some embodiments, the report data can correspond to a plurality ofDICOM images. For example, the report data can include natural languagetext describing a plurality of medical scans of a patient that caninclude multiple sequences, multiple modalities, and/or multiple medicalscans taken over time. In such embodiments, the patient identifying textand/or annotation data detected in the report data can also be appliedto de-identify and/or generate annotation data for the plurality ofDICOM images it describes. In such embodiments, the medical picturearchive integration system 2600 can query the medical picture archivesystem 2620 for one or more additional DICOM images corresponding to thereport data, and de-identified data and annotation data for theseadditional DICOM images can be generated accordingly by utilizing thereport data.

In some embodiments, as shown in FIG. 8E, the medical picture archivesystem 2620 communicates with the report database 2625. The medicalpicture archive system 2620 can request the report data corresponding tothe DICOM image from the report database 2625, and can transmit thereport data to the medical picture archive integration system 2600 via aDICOM communication protocol for receipt via receiver 2602. The medicalpicture archive system 2620 can query the report database 2625 for thereport data, utilizing a common identifier extracted from thecorresponding DICOM image, in response to determining to send thecorresponding DICOM image to the medical picture archive integrationsystem 2600.

FIG. 8F presents an embodiment where report data is generated by theannotating system 2612 and is transmitted, via a transmitter 2605, tothe report database 2625, for example via a DICOM communication protocolor other protocol recognized by the report database 2625. In otherembodiments, the report data is instead transmitted via transmitter 2604to the medical picture archive system 2620, and the medical picturearchive system 2620 transmits the report data to the report database2625.

The report data can be generated by the annotating system 2612 as outputof performing the inference function on the de-identified medical scan.The report data can include natural language text data 448 generatedautomatically based on other diagnosis data 440 such as abnormalityannotation data 442 determined by performing the inference function, forexample, by utilizing a medical scan natural language generatingfunction trained by the medical scan natural language analysis system114. The report data can be generated instead of, or in addition to, theannotated DICOM file.

FIG. 9 presents a flowchart illustrating a method for execution by amedical picture archive integration system 2600 that includes a firstmemory and a second memory that store executable instructions that, whenexecuted by at least one first processor and at least one secondprocessor, respectfully, cause the medical picture archive integrationsystem to perform the steps below. In various embodiments, the firstmemory and at least one first processor are implemented by utilizing,respectfully, the memory 2654 and processing system 2652 of FIG. 8B. Invarious embodiments, the second memory is implemented by utilizing thememory 2674 and/or the memory 2684 of FIG. 8B. In various embodiments,the at least one second processor is implemented by utilizing theprocessing system 2682 of FIG. 8B.

Step 2702 includes receiving, from a medical picture archive system viaa receiver, a first DICOM image for storage in the first memory,designated for PHI, where the first DICOM image includes at least onepatient identifier. Step 2704 includes performing, via at least onefirst processor coupled to the first memory and designated for PHI, ade-identification function on the first DICOM image to identify the atleast one patient identifier and generate a first de-identified medicalscan that does not include the at least one patient identifier.

Step 2706 includes storing the first de-identified medical scan in asecond memory that is separate from the first memory. Step 2708 includesreceiving, via a network interface communicating with a network thatdoes not include the medical picture archive system, first modelparameters from a central server.

Step 2710 includes retrieving the first de-identified medical scan fromthe second memory. Step 2712 includes utilizing the first modelparameters to perform a first inference function on the firstde-identified medical scan to generate first annotation data via atleast one second processor that is different from the at least one firstprocessor. Step 2714 includes generating, via the at least one secondprocessor, a first annotated DICOM file for transmission to the medicalpicture archive system via a transmitter, where the first annotatedDICOM file includes the first annotation data and further includes anidentifier that indicates the first DICOM image. In various embodiments,the first annotated DICOM file is a DICOM presentation state file.

In various embodiments, the second memory further includes operationalinstructions that, when executed by the at least one second processor,further cause the medical picture archive integration system to retrievea second de-identified medical scan from the de-identified image storagesystem, where the second de-identified medical scan was generated by theat least one first processor by performing the de-identificationfunction on a second DICOM image received from the medical picturearchive system. The updated model parameters are utilized to perform thefirst inference function on the second de-identified medical scan togenerate second annotation data. A second annotated DICOM file isgenerated for transmission to the medical picture archive system via thetransmitter, where the second annotated DICOM file includes the secondannotation data and further includes an identifier that indicates thesecond DICOM image.

In various embodiments, the second memory stores a plurality ofde-identified medical scans generated by the at least one firstprocessor by performing the de-identification function on acorresponding plurality of DICOM images received from the medicalpicture archive system via the receiver. The plurality of de-identifiedmedical scans is transmitted to the central server via the networkinterface, and the central server generates the first model parametersby performing a training function on training data that includes theplurality of de-identified medical scans.

In various embodiments, the central server generates the first modelparameters by performing a training function on training data thatincludes a plurality of de-identified medical scans received from aplurality of medical picture archive integration systems via thenetwork. Each of the plurality of medical picture archive integrationsystems communicates bidirectionally with a corresponding one of aplurality of medical picture archive systems, and the plurality ofde-identified medical scans corresponds to a plurality of DICOM imagesstored by the plurality of medical picture archive integration systems.

In various embodiments, the first de-identified medical scan indicates ascan category of the first DICOM image. The second memory further storesoperational instructions that, when executed by the at least one secondprocessor, further cause the medical picture archive integration systemto select the first inference function from a set of inference functionsbased on the scan category. The set of inference functions correspondsto a set of unique scan categories that includes the scan category. Invarious embodiments, each unique scan category of the set of unique scancategories is characterized by one of a plurality of modalities and oneof a plurality of anatomical regions.

In various embodiments, the first memory further stores operationalinstructions that, when executed by the at least one first processor,further cause the medical picture archive integration system to receivea plurality of DICOM image data from the medical picture archive systemvia the receiver for storage in the first memory in response to a querytransmitted to the medical picture archive system via the transmitter.The query is generated by the medical picture archive integration systemin response to a request indicating a new scan category received fromthe central server via the network. The new scan category is notincluded in the set of unique scan categories, and the plurality ofDICOM image data corresponds to the new scan category. Thede-identification function is performed on the plurality of DICOM imagedata to generate a plurality of de-identified medical scans fortransmission to the central server via the network.

The second memory further stores operational instructions that, whenexecuted by the at least one second processor, further cause the medicalpicture archive integration system to receive second model parametersfrom the central server via the network for a new inference functioncorresponding to the new scan category. The set of inference functionsis updated to include the new inference function. The secondde-identified medical scan is retrieved from the first memory, where thesecond de-identified medical scan was generated by the at least onefirst processor by performing the de-identification function on a secondDICOM image received from the medical picture archive system. The newinference function is selected from the set of inference functions bydetermining the second de-identified medical scan indicates the secondDICOM image corresponds to the new scan category. The second modelparameters are utilized to perform the new inference function on thesecond de-identified medical scan to generate second annotation data. Asecond annotated DICOM file is generated for transmission to the medicalpicture archive system via the transmitter, where the second annotatedDICOM file includes the second annotation data and further includes anidentifier that indicates the second DICOM image.

In various embodiments, the medical picture archive integration systemgenerates parameter data for transmission to the medical picture archivesystem that indicates the set of unique scan categories. The medicalpicture archive system automatically transmits the first DICOM image tothe medical picture archive integration system in response todetermining that the first DICOM image compares favorably to one of theset of unique scan categories.

In various embodiments, the second memory further stores operationalinstructions that, when executed by the at least one second processor,cause the medical picture archive integration system to generate anatural language report data is based on the first annotation data andto transmit, via a second transmitter, the natural language report datato a report database associated with the medical picture archiveintegration system, where the natural language report data includes anidentifier corresponding to the first DICOM image.

In various embodiments, the first memory further stores operationalinstructions that, when executed by the at least one first processor,cause the medical picture archive integration system to receive, via asecond receiver, a natural language report corresponding to the firstDICOM image from the report database. A set of patient identifying textincluded in the natural language report are identified. Performing thede-identification function on the first DICOM image includes searchingthe first DICOM image for the set of patient identifying text toidentify the at least one patient identifier.

In various embodiments, the first memory is managed by a medical entityassociated with the medical picture archive system. The medical picturearchive integration system is located at a first geographic sitecorresponding to the medical entity, and the central server is locatedat a second geographic site. In various embodiments, the first memory isdecoupled from the network to prevent the first DICOM image thatincludes the at least one patient identifier from being communicated viathe network. In various embodiments, the medical picture archive systemis a Picture Archive and Communication System (PACS) server, and thefirst DICOM image is received in response to a query sent to the medicalpicture archive system by the transmitter in accordance with a DICOMcommunication protocol.

FIG. 10A presents an embodiment of a de-identification system 2800. Thede-identification system 2800 can be utilized to implement thede-identification system 2608 of FIGS. 8A-8F. In some embodiments, thede-identification system 2800 can be utilized by other subsystems tode-identify image data, medical report data, private fields of medicalscan entries 352 such as patient identifier data 431, and/or otherprivate fields stored in databases of the database memory device 340.

The de-identification system can be operable to receive, from at leastone first entity, a medical scan and a medical report corresponding tothe medical scan. A set of patient identifiers can be identified in asubset of fields of a header of the medical scan. A header anonymizationfunction can be performed on each of the set of patient identifiers togenerate a corresponding set of anonymized fields. A de-identifiedmedical scan can be generated by replacing the subset of fields of theheader of the medical scan with the corresponding set of anonymizedfields.

A subset of patient identifiers of the set of patient identifiers can beidentified in the medical report by searching text of the medical reportfor the set of patient identifiers. A text anonymization function can beperformed on the subset of patient identifiers to generate correspondinganonymized placeholder text for each of the subset of patientidentifiers. A de-identified medical report can be generated byreplacing each of the subset of patient identifiers with thecorresponding anonymized placeholder text. The de-identified medicalscan and the de-identified medical report can be transmitted to a secondentity via a network.

As shown in FIG. 10A, the de-identification system 2800 can include atleast one receiver 2802 operable to receive medical scans, such asmedical scans in a DICOM image format. The at least one receiver 2802 isfurther operable to receive medical reports, such as report data 449 orother reports containing natural language text diagnosing, describing,or otherwise associated the medical scans received by thede-identification system. The medical scans and report data can bereceived from the same or different entity, and can be received by thesame or different receiver 2802 in accordance with the same or differentcommunication protocol. For example, the medical scans can be receivedfrom the medical picture archive system 2620 of FIGS. 8A-8F and thereport data can be received from the report database 2625 of FIGS.8D-8F. In such embodiments, the receiver 2802 can be utilized toimplement the receiver 2602 of FIG. 8B.

The de-identification system 2800 can further include a processingsystem 2804 that includes at least one processor, and a memory 2806. Thememory 2806 can store operational instructions that, when executed bythe processing system, cause the de-identification system to perform atleast one patient identifier detection function on the received medicalscan and/or the medical report to identify a set of patient identifiersin the medical scan and/or the medical report. The operationalinstructions, when executed by the processing system, can further causethe de-identification system to perform an anonymization function on themedical scan and/or the medical report to generate a de-identifiedmedical scan and/or a de-identified medical report that do not includethe set of patient identifiers found in performing the at least onepatient identifier detection function. Generating the de-identifiedmedical scan can include generating a de-identified header andgenerating de-identified image data, where the de-identified medicalscan includes both the de-identified header and the de-identified imagedata. The memory 2806 can be isolated from Internet connectivity, andcan be designated for PHI.

The de-identification system 2800 can further include at least onetransmitter 2808, operable to transmit the de-identified medical scanand de-identified medical report. The de-identified medical scan andde-identified medical report can be transmitted back to the same entityfrom which they were received, respectively, and/or can be transmittedto a separate entity. For example, the at least one transmitter cantransmit the de-identified medical scan to the de-identified imagestorage system 2610 of FIGS. 8A-8F and/or can transmit the de-identifiedmedical scan to central server system 2640 via network 2630 of FIGS.8A-8F. In such embodiments, the transmitter 2808 can be utilized toimplement the interface 2655 of FIG. 8B. The receiver 2802, processingsystem 2804, memory 2806, and/or transmitter 2808 can be connected viabus 2810.

Some or all of the at least one patient identifier detection functionand/or at least one anonymization function as discussed herein can betrained and/or implemented by one or subsystems 101 in the same fashionas other medical scan analysis functions discussed herein, can be storedin medical scan analysis function database 346 of FIG. 3, and/or canotherwise be characterized by some or all fields of a medical scananalysis function entry 356 of FIG. 5.

The de-identification system 2800 can perform separate patientidentifier detection functions on the header of a medical report and/ormedical scan, on the text data of the medical report, and/or on theimage data of the medical scan, such as text extracted from the imagedata of the medical scan. Performance of each of these functionsgenerates an output of its own set of identified patient identifiers.Combining these sets of patient identifiers yields a blacklist term set.A second pass of the header of a medical report and/or medical scan, onthe text data of the medical report, and/or on the image data of themedical scan that utilizes this blacklist term set can catch any termsthat were missed by the respective patient identifier detectionfunction, and thus, the outputs of these multiple identificationprocesses can support each other. For example, some of the data in theheaders will be in a structured form and can thus be easier to reliablyidentify. This can be exploited and used to further anonymize theseidentifiers when they appear in free text header fields, report data,and/or in the image data of the medical scan. Meanwhile, unstructuredtext in free text header fields, report data, and/or image data of themedical scan likely includes pertinent clinical information to bepreserved in the anonymization process, for example, so it can beleveraged by at least one subsystem 101 and/or so it can be leveraged intraining at least one medical scan analysis function.

At least one first patient identifier detection function can includeextracting the data in a subset of fields of a DICOM header, or anotherheader or other metadata of the medical scan and/or medical report witha known type that corresponds to patient identifying data. For example,this patient identifying subset of fields can include a name field, apatient ID number field or other unique patient identifier field, a datefield, a time field, an age field, an accession number field, SOPinstance UID, and/or other fields that could be utilized to identify thepatient and/or contain private information. A non-identifying subset offields of the header can include hospital identifiers, machine modelidentifiers, and/or some or all fields of medical scan entry 352 that donot correspond to patient identifying data. The patient identifyingsubset of fields and the non-identifying subset of fields can bemutually exclusive and collectively exhaustive with respect to theheader. The at least one patient identifier function can includegenerating a first set of patient identifiers by ignoring thenon-identifying subset of fields and extracting the entries of thepatient identifying subset of fields only. This first set of patientidentifiers can be anonymized to generate a de-identified header asdiscussed herein.

In some embodiments, at least one second patient identifier detectionfunction can be performed on the report data of the medical report. Theat least one second patient identifier detection function can includeidentifying patient identifying text in the report data by performing anatural language analysis function, for example, trained by the medicalscan natural language analysis system 114. For example, the at least onesecond patient identifier detection function can leverage the knownstructure of the medical report and/or context of the medical report. Asecond set of patient identifiers corresponding to the patientidentifying text can be determined, and the second set of patientidentifiers can be anonymized to generate a de-identified medicalreport. In some embodiments, a de-identified medical report includesclinical information, for example, because the portion of the originalmedical report that includes the clinical information was deemed to befree of patient identifying text and/or because the portion of theoriginal medical report that includes the clinical information wasdetermined to include pertinent information to be preserved.

In some embodiments, the medical report includes image datacorresponding to freehand or typed text. For example the medical reportcan correspond to a digitized scan of original freehand text written bya radiologist or other medical professional. In such embodiments, thepatient identifier detection function can first extract the text fromthe freehand text in the image data to generate text data before the atleast one second patient identifier detection function is performed onthe text of the medical report to generate the second set of patientidentifiers.

In some embodiments, the at least one second patient identifierdetection function can similarly be utilized to identify patientidentifying text in free text fields and/or unstructured text fields ofa DICOM header and/or other metadata of the medical scan and/or medicalreport data by performing a natural language analysis function, forexample, trained by the medical scan natural language analysis system114. A third set of patient identifiers corresponding to this patientidentifying text of the free text and/or unstructured header fields canbe determined, and the third set of patient identifiers can beanonymized to generate de-identified free text header field and/orunstructured header fields. In some embodiments, a de-identified freetext header field and/or unstructured header field includes clinicalinformation, for example, because the portion of the originalcorresponding header field that includes the clinical information wasdeemed to be free of patient identifying text and/or because the portionof the original corresponding header field that includes the clinicalinformation was determined to include pertinent information to bepreserved.

Patient identifiers can also be included in the image data of themedical scan itself. For example, freehand text corresponding to apatient name written on a hard copy of the medical scan beforedigitizing can be included in the image data, as discussed inconjunction with FIG. 10B. Other patient identifiers, such asinformation included on a patient wristband or other identifyinginformation located on or within the vicinity of the patient may havebeen captured when the medical scan was taken, and can thus be includedin the image. At least one third patient identifier detection functioncan include extracting text from the image data and/or detectingnon-text identifiers in the image data by performing a medical scanimage analysis function, for example, trained by the medical scan imageanalysis system 112. For example, detected text that corresponds to animage location known to include patient identifiers, detected text thatcorresponds to a format of a patient identifier, and/or or detected textor other image data determined to correspond to a patient identifier canbe identified. The at least one third patient identifier detectionfunction can further include identifying patient identifying text in thetext extracted from the image data by performing the at least one secondpatient identifier detection function and/or by performing a naturallanguage analysis function. A fourth set of patient identifierscorresponding to patient identifying text or other patient identifiersdetected in the image data of the medical scan can be determined, andthe fourth set of patient identifiers can be anonymized in the imagedata to generate de-identified image data of the medical scan asdescribed herein. In particular, the fourth set of patient identifierscan be detected in a set of regions of image data of the medical scan,and the set of regions of the image data can be anonymized.

In some embodiments, only a subset of the patient identifier detectionfunctions described herein are performed to generate respective sets ofpatient identifiers for anonymization. In some embodiments, additionalpatient identifier detection functions can be performed on the medicalscan and/or medical report to determine additional respective sets ofpatient identifiers for anonymization. The sets of patient identifiersoutputted by performing each patient identifier detection function canhave a null or non-null intersection. The sets of patient identifiersoutputted by performing each patient identifier function can have nullor non-null set differences.

Cases where the sets of patient identifiers have non-null setdifferences can indicate that a patient identifier detected by onefunction may have been missed by another function. The combined set ofpatient identifiers, for example, generated as the union of the sets ofsets of patient identifiers outputted by performing each patientidentifier function, can be used to build a blacklist term set, forexample, stored in memory 2806. The blacklist term set can designate thefinal set of terms to be anonymized. A second pass of header data,medical scans, medical reports, and/or any free text extracted from theheader data, the medical scan, and/or the medical report can beperformed by utilizing the blacklist term set to flag terms foranonymization that were not caught in performing the respective at leastone patient identifier detection function. For example, performing thesecond pass can include identifying at least one patient identifier ofthe blacklist term set in the header, medical report, and/or image dataof the medical scan. This can include by searching correspondingextracted text of the header, medical report, and/or image data forterms included in blacklist term set and/or by determining if each termin the extracted text is included in the blacklist term set.

In some embodiments, at least one patient identifier is not detecteduntil the second pass is performed. Consider an example where a freetext field of a DICOM header included a patient name that was notdetected in performing a respective patient identifier detectionfunction on the free text field of the DICOM header. However, thepatient name was successfully identified in the text of the medicalreport in performing a patient identifier detection function on themedical report. This patient name is added to the blacklist term list,and is detected in a second pass of the free text field of the DICOMheader. In response to detection in the second pass, the patient name ofthe free text field of the DICOM header can be anonymized accordingly togenerate a de-identified free text field. Consider a further examplewhere the patient name is included in the image data of the medicalscan, but was not detected in performing a respective patient identifierdetection function on the free text field of the DICOM header. In thesecond pass, this patient name can be detected in at least one region ofimage data of the medical scan by searching the image data for theblacklist term set.

In some embodiments, performing some or all of the patient identifierdetection functions includes identifying a set of non-identifying terms,such as the non-identifying subset of fields of the header. Inparticular, the non-identifying terms can include terms identified asclinical information and/or other terms determined to be preserved. Thecombined set of non-identifying terms, for example, generated as theunion of the sets of sets of non-identifying outputted by performingeach patient identifier function, can be used to build a whitelist termset, for example, stored in memory 2806. Performing the second pass canfurther include identifying at least one non-identifying term of thewhitelist term set in the header, medical report, and/or image data ofthe medical scan, and determining not to anonymize, or to otherwiseignore, the non-identifying term.

In various embodiments, some or all terms of the whitelist term set canbe removed from the blacklist term set. In particular, at least one termpreviously identified as a patient identifier in performing one or morepatient identifier detection functions is determined to be ignored andnot anonymized in response to determining the term is included in thewhitelist term set. This can help ensure that clinically importantinformation is not anonymized, and is thus preserved in thede-identified medical scan and de-identified medical report.

In some embodiments, the second pass can be performed after each of thepatient identifier detection functions are performed. For example,performing the anonymization function can include performing this secondpass by utilizing the blacklist term set to determine the final set ofterms to be anonymized. New portions of text in header fields, notpreviously detected in generating the first set of patient identifiersor the third set of patient identifiers, can be flagged foranonymization by determining these new portions of text correspond toterms of the blacklist term set. New portions of text the medicalreport, not previously detected in generating in the second set ofpatient identifiers, can be flagged for anonymization by determiningthese new portions of text correspond to terms of the blacklist termset. New regions of the image data of the medical scan, not previouslydetected in generating the fourth set of patient identifiers, can beflagged for anonymization by determining these new portions of textcorrespond to terms of the blacklist term set.

In some embodiments, the blacklist term set is built as each patientidentifier detection function is performed, and performance ofsubsequent patient identifier detection functions includes utilizing thecurrent blacklist term set. For example, performing the second patientidentifier detection function can include identifying a first subset ofthe blacklist term set in the medical report by searching the text ofthe medical report for the blacklist term set and/or by determining ifeach term in the text of the medical report is included in the blacklistterm set. Performing the second patient identifier detection functioncan further include identifying at least one term in the medical reportthat is included in the whitelist term set, and determining to ignorethe term in response. The first subset can be anonymized to generate thede-identified medical report as discussed herein. New patientidentifiers not already found can be appended to the blacklist term set,and the updated blacklist term set can be applied to perform a secondsearch of the header and/or image data of the medical scan, and at leastone of the new patient identifiers can be identified in the header inthe second search of the header and/or in the image data in a secondsearch of the image data. These newly identified patient identifiers inthe header and/or image data are anonymized in generating thede-identified medical scan.

As another example, a second subset of the blacklist term set can bedetected in a set of regions of image data of the medical scan byperforming the medical scan image analysis function on image data of themedical scan, where the image analysis function includes searching theimage data for the set of patient identifiers. For example, the medicalscan image analysis function can include searching the image data fortext, and the second subset can include detected text that matches oneor more terms of the blacklist term set. In some embodiments, detectedtext that matches one or more terms of the whitelist term set can beignored. The second subset can be anonymized to generate de-identifiedimage data as discussed herein. New patient identifiers that aredetected can be appended to the blacklist term set, and the updatedblacklist term set can be applied to perform a second search of theheader and/or metadata of the medical scan, and/or can be applied toperform a second search of the medical report. At least one of the newpatient identifiers can be identified in the header as a result ofperforming the second search of the header and/or at least one of thenew patient identifiers can be identified medical report as a result ofperforming the second search of the medical report. These newlyidentified patient identifiers can be anonymized in the header alongwith the originally identified blacklist term set in generating thede-identified header, and/or can be anonymized in the medical reportalong with the originally identified first subset in generating thede-identified medical report.

In some embodiments, the memory 2806 further stores a global blacklist,for example, that includes a vast set of known patient identifyingterms. In some embodiments, the global blacklist is also utilized by atleast one patient identifier detection function and/or in performing thesecond pass to determine patient identifying terms for anonymization. Insome embodiments, the blacklist term set generated for a particularmedical scan and corresponding medical report can be appended to theglobal blacklist for use in performing the second pass and/or indetecting patient identifiers in subsequently received medical scansand/or medical reports.

Alternatively or in addition, the memory 2806 can further store a globalwhitelist, for example, that includes a vast set of terms that can beignored. In particular, the global whitelist can include clinical termsand/or other terms that are deemed beneficial to preserve that do notcorrespond to patient identifying information. In some embodiments, theglobal whitelist is utilized by at least one patient identifierdetection function and/or in performing the second pass to determineterms to ignore in the header, image data, and/or medical report. Insome embodiments, the whitelist term set generated for a particularmedical scan and corresponding medical report can be appended to theglobal whitelist for use in performing the second pass and/or inignoring terms in subsequently received medical scans and/or medicalreports.

Alternatively or in addition, the memory 2806 can further store a globalgraylist, for example, that includes ambiguous terms that could bepatient identifying terms in some contexts, but non-identifying terms inother contexts. For example, “Parkinson” could correspond to patientidentifying data if part of a patient name such as “John Parkinson”, butcould correspond to non-patient identifying data meant to be ignored andpreserved in the de-identified medical report and/or de-identifiedmedical scan if part of a diagnosis term such as “Parkinson's disease.”In some embodiments, the global graylist is also utilized in performingthe second pass and/or in performing at least one patient identifierdetection function to determine that a term is included in the graylist,and to further determine whether the term should be added to theblacklist term set for anonymization or whitelist term set to be ignoredby leveraging context of accompanying text, by leveraging known datatypes of a header field from which the term was extracted, by leveragingknown structure of the term, by leveraging known data types of alocation of the image data from which the term was extracted, and/or byleveraging other contextual information. In some embodiments, thegraylist term set can be updated based on blacklist and/or whitelistterm sets for a particular medical scan and corresponding medicalreport.

In some embodiments, the at least one anonymization function includes afiducial replacement function. For example, some or all of the blacklistterm set can be replaced with a corresponding, global fiducial in theheader, report data, and/or image data. In some embodiments, the globalfiducial can be selected from a set of global fiducials based on a typeof the corresponding patient identifier. Each patient identifierdetected in the header and/or medical report can be replaced with acorresponding one of the set of global text fiducials. Each patientidentifiers detected in the image data can be replaced with acorresponding one of the set of global image fiducials. For example, oneor more global image fiducials can overlay pixels of regions of theimage data that include the identifying patient data, to obfuscate theidentifying patient data in the de-identified image data.

The global text fiducials and/or global image fiducials can berecognizable by inference functions and/or training functions, forexample, where the global text fiducials and global image fiducials areignored when processed in a training step to train an inference functionand/or are ignored in an inference step when processed by an inferencefunction. Furthermore, the global text fiducials and/or global imagefiducials can be recognizable by a human viewing the header, medicalreport, and/or image data. For example, a radiologist or other medicalprofessional, upon viewing a header, medical report, and/or image data,can clearly identify the location of a patient identifier that wasreplaced by the fiducial and/or can identify the type of patientidentifier that was replaced by the fiducial.

As an example, the name “John Smith” can be replaced in a header and/ormedical report with the text “%PATIENT NAME%”, where the text “%PATIENTNAME%” is a global fiducial for name types of the header and/or the textof medical reports. The training step and/or inference step of medicalscan natural language analysis functions can recognize and ignore textthat matches “%PATIENT NAME%” automatically.

FIG. 10B illustrates an example of anonymizing patient identifiers inimage data of a medical scan. In this example, the name “John Smith” andthe date “May 4, 2010” is detected as freehand text in the originalimage data of a medical scan. The regions of the image data that includethe patient identifiers can each be replaced by global fiducial in theshape of a rectangular bar, or any other shape. As shown in FIG. 10B, afirst region corresponding to the location of “John Smith” in theoriginal image data is replaced by fiducial 2820 in the de-identifiedimage data, and a second region corresponding to the location of “May 4,2010” in the original image data is replaced by fiducial 2822 in thede-identified image data. The size, shape, and/or location of eachglobal visual fiducial can be automatically determined based on thesize, shape, and/or location of the region that includes the patientidentifier to minimize the amount of the image data that is obfuscated,while still ensuring the entirety of the text is covered. While notdepicted in FIG. 10B, the fiducial can be of a particular color, forexample, where pixels of the particular color are automaticallyrecognized by the training step and/or inference step of medical scanimage analysis functions to indicate that the corresponding region beignored, and/or where the particular color is not included in theoriginal medical scan and/or is known to not be included in any medicalscans. The fiducial can include text recognizable to human inspectionsuch as “%PATIENT NAME” and “%DATE” as depicted in FIG. 10B, and/or caninclude a QR code, logo, or other unique symbol recognizable to humaninspection and/or automatically recognizable by the training step and/orinference step of medical scan image analysis functions to indicate thatthe corresponding region be ignored.

In some embodiments, other anonymization functions can be performed ondifferent ones of the patient identifying subset of fields to generatethe de-identified header, de-identified report data, and/orde-identified image data. For example, based on the type of identifyingdata of each field of the header, different types of headeranonymization functions and/or text anonymization functions can beselected and utilized on the header fields, text of the report, and/ortext extracted from the image data. A set of anonymization functions caninclude a shift function, for example, utilized to offset a date, timeor other temporal data by a determined amount to preserve absolute timedifference and/or to preserve relative order over multiple medical scansand/or medical reports of a single patient. FIG. 10B depicts an examplewhere the shift function is performed on the date detected in the imagedata to generate fiducial 2822, where the determined amount is 10 yearsand 1 month. The determined amount can be determined by thede-identification system randomly and/or pseudo-randomly for eachpatient and/or for each medical scan and corresponding medical report,ensuring the original date cannot be recovered by utilizing a knownoffset. In various embodiments, other medical scans and/or medicalreports are fetched for the same patient by utilizing a patient IDnumber or other unique patient identifier of the header. These medialscans and reports can be anonymized as well, where the dates and/ortimes detected in these medical scans and/or medical reports offset bythe same determined amount, randomized or pseudo-randomized forparticular patient ID number, for example, based on performing a hashfunction on the patient ID number.

The set of anonymization functions can include at least one hashfunction, for example utilized to hash a unique patient ID such as apatient ID number, accession number, and/or SOP instance UID of theheader and/or text. In some embodiments, the hashed SOP instance UID,accession number, and/or patient ID number are prepended with a uniqueidentifier, stored in a database of the memory 2806 and/or shared withthe entities to which the de-identified medical scans and/or medicalreports are transmitted, so that de-identified medical scans and theircorresponding de-identified medical reports can be linked and retrievedretroactively. Similarly, longitudinal data can be preserved as multiplemedical scans and/or medical reports of the same patient will beassigned the same hashed patient ID.

The set of anonymization functions can further include at least onemanipulator function for some types of patient identifiers. Some valuesof header fields and/or report text that would normally not beconsidered private information can be considered identifying patientdata if they correspond to an outlier value or other rare value thatcould then be utilized to identify the corresponding patient from a verysmall subset of possible options. For example, a patient age over 89could be utilized to determine the identity of the patient, for example,if there are very few patients over the age of 89. To prevent suchcases, in response to determining that a patient identifier correspondsto an outlier value and/or in response to determining that a patientidentifier compares unfavorably to a normal-range threshold value, thepatient identifier can be capped at the normal-range threshold value orcan otherwise be manipulated. For example, a normal-range thresholdvalue corresponding to age can be set at 89, and generating ade-identified patient age can include capping patient ages that arehigher than 89 at 89 and/or can include keeping the same value forpatient ages that are less than or equal to 89.

In some embodiments, the de-identified header data is utilized toreplace the corresponding first subset of patient identifiers detectedin the medical report with text of the de-identified header fields. Inother embodiments, a set of text anonymization functions includes aglobal text fiducial replacement function, shift function, a hashfunction, and/or manipulator functions that anonymize the correspondingtypes of patient identifiers in the medical report separately.

In some embodiments where the image data of a medical scan includes ananatomical region corresponding to a patient's head, the image data mayinclude an identifying facial structure and/or facial features thatcould be utilized to determine the patient's identity. For example, adatabase of facial images, mapped to a corresponding plurality of peopleincluding the patient, could be searched and a facial recognitionfunction could be utilized to identify the patient in the database.Thus, facial structure included in the image data can be consideredpatient identifying data.

To prevent this problem and maintain patient privacy, thede-identification system can further be implemented to perform facialobfuscation for facial structure detected in medical scans. At least oneregion of the image data that includes identifying facial structure canbe determined by utilizing a medical image analysis function. Forexample, the medical image analysis function can include a facialdetection function that determines the regions of the image data thatinclude identifying facial structure based on searching the image datafor pixels with a density value that corresponds to facial skin, facialbone structure, or other density of an anatomical mass type thatcorresponds to identifying facial structure, and the facial obfuscationfunction can be performed on the identified pixels. Alternatively or inaddition, the facial detection function can determine the region basedon identifying at least one shape in the image data that corresponds toa facial structure.

The image obfuscation function can include a facial structureobfuscation function performed on the medical scan to generatede-identified image data that does not include identifying facialstructure. For example, the facial structure obfuscation function canmask, scramble, replace with a fiducial, or otherwise obfuscate thepixels of the region identified by the facial detection function. Insome embodiments, the facial structure obfuscation function can performa one-way function on the region that preserves abnormalities of thecorresponding portions of the image, such as nose fractures or facialskin legions, while still obfuscating the identifying facial structuresuch that the patient is not identifiable. For example, the pixels ofthe identifying facial structure can be altered such that they convergetowards a fixed, generic facial structure. In some embodiments, aplurality of facial structure image data of a plurality of patients canbe utilized to generate the generic facial structure, for example,corresponding to an average or other combination of the plurality offaces. For example, the pixels of the generic facial structure can beaveraged with, superimposed upon, or otherwise combined with the pixelsof the region of the image data identified by the facial detectionfunction in generating the de-identified image data.

In some embodiments, a hash function can be performed on an average ofthe generic facial structure and the identified facial structure of theimage data so that the generic facial structure cannot be utilized inconjunction with the resulting data of the de-identified image data toreproduce the original, identifying facial structure. In suchembodiments, the hash function can alter the pixel values while stillpreserving abnormalities. In some embodiments, a plurality of random,generic facial structures can be generated by utilizing the plurality offacial structure image data, for example, where each if the plurality offacial structure image data are assigned a random or pseudo-randomweight in an averaging function utilized to create the generic facialstructure, where a new, random or pseudo-random set of weights aregenerated each time the facial structure obfuscation function isutilized to create a new, generic facial structure to be averaged withthe identified facial structure in creating the de-identified image datato ensure the original identifying facial structure cannot be extractedfrom the resulting de-identified image data.

While facial obfuscation is described herein, similar techniques can beapplied in a similar fashion to other anatomical regions that aredetermined to include patient identifiers and/or to other anatomicalregions that can be utilized to extract patient identifying informationif not anonymized.

In some embodiments, the at least one receiver 2802 is included in atleast one transceiver, for example, enabling bidirectional communicationbetween the medical picture archive system 2620 and/or the reportdatabase 2625. In such embodiments, the de-identification system 2800can generate queries to the medical picture archive system 2620 and/orthe report database 2625 for particular medical scans and/or medicalreports, respectively. In particular, if the medical scan and medicalreport are stored and/or managed by separate memories and/or separateentities, they may not be received at the same time. However, a linkingidentifier, such as DICOM identifiers in headers or metadata of themedical scan and/or medical report, such accession number, patient IDnumber, SOP instance UID, or other linking identifier that maps themedical scan to the medical report can be utilized to fetch a medicalreport corresponding to a received medical scan and/or to fetch amedical scan corresponding to a received medical report via a query sentutilizing the at least one transceiver. For example, in response toreceiving the medical scan from the medical picture archive system 2620,the de-identification system can extract a linking identifier from aDICOM header of the medical scan, and can query the report database 2625for the corresponding medical report by indicating the linkingidentifier in the query. Conversely, in response to receiving themedical report from the report database 2625, the de-identificationsystem can extract the linking identifier from a header, metadata,and/or text body of the medical report, and can query the medicalpicture archive system 2620 for the corresponding medical scan byindicating the linking identifier in the query. In some embodiments, amapping of de-identified medical scans to original medical scans, and/ora mapping of de-identified medical reports to original medical reportscan be stored in memory 2806. In some embodiments, linking identifierssuch as patient ID numbers can be utilized to fetch additional medicalscans, additional medical reports, or other longitudinal datacorresponding to the same patient.

FIG. 11 presents a flowchart illustrating a method for execution by ade-identification system 2800 that stores executable instructions that,when executed by at least one processor, cause the de-identification toperform the steps below.

Step 2902 includes receiving from a first entity, via a receiver, afirst medical scan and a medical report corresponding to the medicalscan. Step 2904 includes identifying a set of patient identifiers in asubset of fields of a first header of the first medical scan. Step 2906includes performing a header anonymization function on each of the setof patient identifiers to generate a corresponding set of anonymizedfields. Step 2908 includes generating a first de-identified medical scanby replacing the subset of fields of the first header of the firstmedical scan with the corresponding set of anonymized fields. Step 2910includes identifying a first subset of patient identifiers of the set ofpatient identifiers in the medical report by searching text of themedical report for the set of patient identifiers. Step 2912 includesperforming a text anonymization function on the first subset of patientidentifiers to generate corresponding anonymized placeholder text foreach of the first subset of patient identifiers. Step 2914 includesgenerating a de-identified medical report by replacing each of the firstsubset of patient identifiers with the corresponding anonymizedplaceholder text. Step 2916 includes transmitting, via a transmitter,the de-identified first medical scan and the de-identified medicalreport to a second entity via a network.

In various embodiments, the medical scan is received from a PictureArchive and Communication System (PACS), where the medical report isreceived from a Radiology Information System (RIS), and where the firstde-identified medical scan and the de-identified medical report aretransmitted to a central server that is not affiliated with the PACS orthe RIS. In various embodiments, first medical scan and the medicalreport are stored in a first memory for processing. The first memory isdecoupled from the network to prevent the set of patient identifiersfrom being communicated via the network. The first de-identified medicalscan and the de-identified medical report are stored in a second memorythat is separate from the first memory. The first de-identified medicalscan and the de-identified medical report are fetched from the secondmemory for transmission to the second entity.

In various embodiments, the header anonymization function performed oneach of the set of patient identifiers is selected from a plurality ofheader anonymization functions based on one of a plurality of identifiertypes of the corresponding one of the subset of fields. In variousembodiments, the plurality of identifier types includes a date type. Ashift function corresponding to the date type is performed on a firstdate of the first header to generate the first de-identified medicalscan, where the shift function includes offsetting the first date by adetermined amount. A second medical scan is received, via the receiver,that includes a second header. A unique patient ID of the first headermatches a unique patient ID of the second header. The shift function isperformed on a second date of the second header by offsetting the seconddate by the determined amount to generate a second de-identified medicalscan. The second de-identified medical scan is transmitted to the secondentity via the network.

In various embodiments, the plurality of identifier types includes aunique patient ID type. A hash function corresponding the unique patientID type is performed on the unique patient ID of the first header togenerate the first de-identified medical scan. The hash function isperformed on the unique patient ID of the second header to generate thesecond de-identified medical scan. An anonymized unique patient ID fieldof the first de-identified medical scan matches an anonymized uniquepatient ID field of the second de-identified medical scan as a result ofthe unique patient ID of the first header matching the unique patient IDof the second header.

In various embodiments, the plurality of identifier types includes alinking identifier type that maps the medical scan to the medicalreport. A hash function corresponding to the linking identifier type isperformed on a linking identifier of the first header to generate ahashed linking identifier. A linking identifier field of the firstde-identified medical scan includes the hashed linking identifier.Performing the text anonymization function on the first subset ofpatient identifiers includes determining one of the first subset ofpatient identifiers corresponds to linking identifier text andperforming the hash function on the one of the first subset of patientidentifiers to generate the hashed linking identifier, where thede-identified medical report includes the hashed linking identifier.

In various embodiments, a second subset of patient identifiers of theset of patient identifiers is identified in a set of regions of imagedata of the medical scan by performing an image analysis function onimage data of the medical scan. The image analysis function includessearching the image data for the set of patient identifiers. Anidentifier type is determined for each of the second subset of patientidentifiers. One of a plurality of image fiducials is selected for eachof the second subset of patient identifiers based on the identifiertype. De-identified image data is generated, where a set of regions ofthe de-identified image data, corresponding to the set of regions of theimage data, includes the one of the plurality of image fiducials toobfuscate each of the second subset of patient identifiers. Generatingthe first de-identified medical scan further includes replacing theimage data of the medical scan with the de-identified image data.

In various embodiments, a new patient identifier is identified in themedical report by performing a natural language analysis function on themedical report, where new patient identifier is not included in the setof patient identifiers. The set of patient identifiers is updated toinclude the new patient identifier prior to searching the image data ofthe medical scan for the set of patient identifiers, and the secondsubset of patient identifiers includes the new patient identifier.

In various embodiments, the memory further stores a global identifierblacklist. The natural language analysis function includes searching themedical report for a plurality of terms included in the globalidentifier blacklist to identify the new patient identifier. In variousembodiments, the de-identification system determines that the globalidentifier blacklist does not include one of the set of patientidentifiers, and the global identifier blacklist is updated to includethe one of the set of patient identifiers.

In various embodiments, performing the image analysis function furtherincludes identifying a new patient identifier in the image data, wherenew patient identifier is not included in the set of patientidentifiers. Identifying text is extracted from a region of the imagedata corresponding to the new patient identifier. The new patientidentifier is identified in the medical report by searching text of themedical report for the identifying text. The text anonymization functionis performed on new patient identifier to generate anonymizedplaceholder text for the new patient identifier. Generating thede-identified medical report further includes replacing the identifyingtext with the anonymized placeholder text for the new patientidentifier.

In various embodiments, generating the de-identified image data furtherincludes detecting an identifying facial structure in the image data ofthe medical scan. Generating the de-identified image data includesperforming a facial structure obfuscation function on the image data,and where the de-identified image data does not include the identifyingfacial structure.

In some embodiments, the medical scans discussed herein can includeelectrocardiogram (ECG) data indicating signal data for one or moreleads corresponding to recorded electrical activity of the heart of apatient and/or other heart beat and/or heart rhythm information of apatient. This can include ECG data and/or EKG data. As used herein, ECGcan interchangeably refer to EKG. Electrocardiogram data correspondingto data recorded by an ECG machine can be processed instead of or inaddition to medical scans as discussed in the various embodiments of themedical scan processing system 100 discussed herein. For example, thesame or different medical scan database 342 can includes ECG entries, asdiscussed in further detail in conjunction with FIG. 12B. One or moreinference functions or other medical scan analysis functions discussedherein can utilize ECG signal data instead of or in addition to othermedical scans as input and can generate diagnosis data as output.

FIG. 12A illustrates an ECG interpretation system 4002 that is operableto process captured images of ECG printouts to automatically generatediagnosis data. In particular, the ECG interpretation system 4002 isoperable to receive captured images of ECG printouts, taken by an imagecapture devices 4012 such as a scanner or camera. An ECG printout cancorrespond to an ECG paper printout generated based on raw signal datarecorded by a ECG machine. The ECG interpretation system 4002 enables auser of client device 120, such as a patient or a medical professionalproviding care for a patient, to receive automatically generateddiagnosis data for an ECG printout. The user can send a captured imageof the ECG printout to the ECG interpretation system 4002 via network150. For example, as illustrated in FIG. 12A, the image capture device4012 can be coupled to or can otherwise communicate with the clientdevice 120. Alternatively, a captured image of an ECG printout can beuploaded to or can otherwise be received by the client device 120, forexample, via the network 150, before transmission to the ECGinterpretation system 4002. In some embodiments, the captured imagecorresponds to a digital ECG file that includes the raw signal datacaptured by the ECG machine, utilized by the ECG machine to generate adigital ECG file directly. The captured image can include any image dataor signal data indicating at least one waveform generated by an ECGmachine. As illustrated in FIG. 12A, multiple client devices 1-Ncorresponding to multiple users can transmit captured images to the ECGinterpretation system 4002.

The ECG interpretation system can process a captured image by performingan ECG processing function 4005 to generate diagnosis data for the ECGprintout. This can include reconstructing pseudo-raw signal data formultiple waveforms detected in the captured image, for example,corresponding to multiple ECG leads. The reconstructed pseudo-raw signaldata can be utilized to generate diagnosis data by utilizing one or moreneural network models and/or other machine learning models trained onECG signal data as input label data and known ECG diagnosis data asoutput label data. The ECG processing function 4005 is discussed furtherin conjunction with FIGS. 12C-12E. Diagnosis data and/or thereconstructed pseudo-raw signal data generated can be transmitted to thecorresponding client device for display via a user interface 4075. Asillustrated in FIG. 12A, diagnosis data 1-N is generated for receivedcaptured images 1-N for transmission to the corresponding client devices1-N from which the corresponding captured image was received. While notdepicted in FIG. 12A, the diagnosis data and/or the reconstructedpseudo-raw signal data can be transmitted to the a client device 120that is different from the client device that transmitted the capturedimage, where the different client device 120 corresponds to the same ordifferent user.

Alternatively or in addition, the diagnosis data, reconstructedpseudo-raw signal data, and/or the captured image can also betransmitted to an ECG database 4342. The ECG database can be implementedby utilizing medical scan database 342, where the captured image datacorresponds to image data 410 and/or where the diagnosis datacorresponds to diagnosis data 440. The ECG database 4342 can be adifferent database of database storage system 140, and can communicatewith some or all other subsystems 101. The ECG database 4342 can store aplurality of ECG entries, where some or all entries include capturedimages of ECG printouts generated by client devices 120 and/or diagnosisdata generated by the ECG interpretation system 4002. Alternately or inaddition, some or all ECG entries can correspond to other ECG printoutsand/or raw signal data of other ECGs. These ECG entries can includediagnosis data generated by a medical professional. For example, suchECG entries can correspond to training data utilized by the ECGinterpretation system 4002 or other system to train at least oneinference function utilized in performing the ECG processing function4005.

As shown in FIG. 12A, the ECG interpretation system 4002 can communicatebidirectionally, via network 150, with the ECG database 4342 and/or withother databases of the database storage system 140, with one or moreclient devices 120, and/or, while not shown in FIG. 12A, with one ormore subsystems 101 of FIG. 1, with the central server system 2640,and/or with the medical picture archive system 2600.

In some embodiments, the ECG interpretation system 4002 is an additionalsubsystem 101 of the medical scan processing system 100, implemented byutilizing the subsystem memory device 245, subsystem processing device235, and/or subsystem network interface 265 of FIG. 2A. For example, theECG interpretation system 4002 can be implemented by utilizing themedical scan image analysis system 112 to train one or more computervision models utilized to perform the ECG processing function 4005and/or by utilizing the medical scan assisted review system 102 topresent, via user interface 4075, diagnosis data in conjunction with theECG printout in the captured image and/or in conjunction with avisualization of reconstructed pseudo-raw signal data. The ECGinterpretation system 4002 can be implemented by utilizing any of thefeatures discussed in conjunction with the captured image processingsystem of FIGS. 15A-15C.

In some embodiments, the ECG interpretation system 4002 utilizes, orotherwise communicates with, the central server system 2640. Forexample, the ECG database 4342 can be populated with de-identified datagenerated by the medical picture archive integration system 2600, forexample, where the medical picture archive integration system 2600 isoperable to generate de-identified ECGs from captured images of ECGprintouts or from digitized ECG files. The ECG interpretation system4002 can receive de-identified ECGs in training sets and/or fordiagnosis with their diagnosis data and/or other relevant informationdirectly from the medical picture archive integration system 2600, forexample, where the diagnosis data is utilized to determine the medicallabels for medical scans in the training set. As another example, theECG interpretation system 4002 can perform one or more inferencefunctions of the ECG processing function 4005 on de-identified ECGs thatneed to be diagnosed, received from the medical picture archiveintegration system 2600. The diagnosis data generated as output can beassigned to the medical scan in the medical picture archive integrationsystem 2600. As another example, the ECG interpretation system 4002 canrequest de-identified ECGs and/or other corresponding information thatmatch requested criteria for a training set utilized to train the ECGprocessing function 4005 and/or for new ECGs to be diagnosed. As anotherexample, the ECG processing function 4005 can be utilized by theannotating system 2612 to generate diagnosis data for ECGs. In someembodiments, some or all of the ECG interpretation system 4002 isimplemented by utilizing other subsystems 101 and/or is operable toperform functions or other operations described in conjunction with oneor more other subsystems 101.

While FIG. 12A depicts communication between the client device and ECGinterpretation system 4002 via network 150, in other embodiments, theclient device can store application data for some or all of the ECGprocessing function 4005. For example, one or more client devices 120 ofFIG. 12A can locally execute the application data to execute ECGprocessing function 4005 locally on captured images taken by or uploadedto the client device and to generate diagnosis data automatically inresponse.

FIG. 12B illustrates an example ECG entry 4452 of ECG database 4342.Alternatively or in addition, some or all fields of the ECG entry 4342can correspond to data that can be transmitted in conjunction withcaptured image data of an ECG printout and/or diagnosis data, torepresent data that can be generated in conjunction with generatingdiagnosis data for an ECG, and/or to represent data that can be utilizedin conjunction with the captured image data of an ECG printout togenerate the diagnosis data.

The ECG entry 4452 can include an ECG identifier 4453 that is unique tothe entry and/or can be utilized to identify the ECG. The ECG identifier4453 can correspond to the same or different type of identifier asmedical scan entry identifier 353. The ECG entry 4452 can include ECHsignal data 4410, which can include a plurality of voltage v. timecoordinates 4415 characterizing the ECG signal of one of a plurality ofECG leads 4412. For example, the ECG signal data 4410 can include datacharacterizing waveforms for each of twelve ECG leads and/or a differentnumber of ECG leads. Some or all of the signal data for some or all ECGleads 4412 can instead include other coordinate data, signal parameterdata, waveform measurements such as some or all of measurement data4443, and/or other graphical waveform data characterizing thecorresponding signal. The ECG signal data 4410 can correspond to rawsignal data, for example, captured directly by an ECG machine.Alternatively, the ECG signal data 4410 can correspond to reconstructed,pseudo-raw signal data, for example, generated by the ECG interpretationsystem 4002 from a captured image of an ECG printout.

The ECG entry can include ECG printout data 4416. The ECG printout data4416 can include a captured image 4417, for example, corresponding tothe captured image generated by client device 120 as discussed inconjunction with FIG. 12A. An ECG printout depicted in the capturedimage can correspond to a physical, paper printout of an ECG generatedto reflect raw signal data recorded by a ECG machine. Alternatively orin addition, an ECG printout depicted in the captured image can insteadcorrespond to a picture taken of an ECG monitor of an ECG machine,displaying raw signal data recorded by the ECG machine. Alternatively orin addition, an ECG printout depicted in the captured image cancorrespond to a picture taken of an ECG monitor of an ECG machine,displaying raw signal data recorded by the ECG machine. Alternatively orin addition, the ECG printout can correspond to a screenshot of areal-time display of raw signal data recorded by the ECG machine on anECG monitor or other display device. The captured image can be stored ina JPEG, PNG, PDF, or other file format utilized to store image data.Alternatively or in addition, the ECG printout can correspond to a videoof a real-time display of raw signal data recorded by the ECG machine onan ECG monitor or other display device, where the captured imagecorresponds to a video file.

The ECG printout data 4416 can include a receiving entity identifier4418 indicating an identifier of the client device 120 from which theECG printout was received by the ECG database 4342 and/or ECGinterpretation system 4002, indicating an identifier of a user accountfrom whom the ECG printout was sent to the ECG database 4342 and/or theECG interpretation system 4002, and/or another identifier indicating anorigin of the captured image. The ECG printout data 4416 can includeadditional details 4419 for the captured image, for example, included inthe transmission of the captured image based on user input to userinterface 4075 as discussed in FIGS. 13A-13C and/or automaticallyextracted by the ECG interpretation system 4002 from text or otherinformation identified in the ECG printout. User account data associatedwith receiving entity identifier 4418 and/or the additional details 4419can be utilized to determine some or all of the ECG classifier data4420, patient history data 4430, diagnosis data 4440, diagnosis authordata 4450, and/or display parameter data 4470.

The ECG entry can include ECG classifier data 4420, which can correspondto some or all of the medical scan classifier data 420. Some or all ofthe ECG scan classifier data can be determined based on additionaldetails 4419 and/or determined based on a medical entity associated withclient device 120. ECG classifier data 4420 can include originatingentity data 4423, which can be the same or different from originatingentity data 423, and/or can indicate a hospital or other medical entityfrom which the original ECG printout was generated. ECG classifier data4420 can include geographic region data 4424 entity data 4423, which canbe the same or different from geographic region data 424, and/or canindicate a geographic from which the original ECG printout was generatedand/or from which a patient for whom the ECG corresponds to lives and/oris otherwise associated with. ECG classifier data 4420 can includemachine data 4425, which can be the same or different from machine data425, and/or can indicate the ECG machine model and/or ECG machineparameters corresponding to the ECG machine from which the original ECGprintout was generated. ECG classifier data 4420 can include ECG datedata 4426, which can be the same or different from scan date data 426,and/or can indicate a date, time, and/or other temporal datacorresponding to when the ECG signals of the ECG printout were recorded.ECG classifier data 4420 can include ECG priority data 4427, which canbe the same or different from scan priority data 427, and/or canindicate a priority value and/or queue value utilized to indicate atime-sensitivity for review and/or follow-up of the ECG, for example,based on the diagnosis data and/or based on the ECG date data. The ECGclassifier data 4420 can include additional classifier data providingadditional details regarding the ECG printout itself.

The ECG entry can include patient history data 4430, which can includepatient identifier data 4431 identifying a patient of whom the ECG wasrecorded, which can be the same or different type of identifier aspatient identifier 431. The patient history data 4430 can include riskfactor data 4432 for the patient, which can same or different riskfactors as risk factor data 432. Some or all of the risk factor data4432 can include risk factors associated with cardiac-based diagnosesand/or risk factors that would aid in generating diagnosis data inconjunction with ECG data. The patient history data 4430 can includelongitudinal data 4433, which can include one or more additional ECGsand/or one or more medical scans associated with the patient. Thelongitudinal quality score 4434 can be based on a quantity and/or timebetween some or all longitudinal data 4433 and/or can be based on thesame or different factors as longitudinal quality score 434.

The ECG entry can include diagnosis data 4440. Some or all ECG entriescan include diagnosis data 4440 generated by the ECG interpretationsystem 4002. Some or all ECG entries can include diagnosis data 4440generated by human review of the ECG printout and/or by human review ofraw signal data and/or pseudo-raw signal data of the ECG signal data4410. The diagnosis data 4440 can include a binary abnormalityidentifier 4441 indicating whether or not the ECG is normal.

The diagnosis data 4440 can include measurement data 4443 indicating oneor more measurements. The measurement data 4443 can include P wave dataindicating time and/or voltage coordinate data of the P wave in the ECGsignal data and/or corresponding measurements and/or characterization ofthe P wave. The measurement data 4443 can include QRS complex dataindicating the Q wave, the R wave, and/or the S wave. The QRS complexdata can include time and/or voltage coordinate data of the Q, R and/orS wave in the ECG signal data and/or can include correspondingmeasurements of the Q, R and/or S wave. The measurement data 4443 caninclude T wave data indicating time and/or voltage coordinate data ofthe T wave in the ECG signal data and/or corresponding measurementsand/or characterization of the T wave. The measurement data 4443 caninclude a measurement for the PR interval, a measurement for the QRSduration, and/or a measurement for the QT interval. The measurement data4443 can include a heart rate approximation. Some or all of themeasurement data can be generated by the ECG interpretation system 4002and/or can be extracted from text included in the ECG printout. Themeasurement data 4443 can further include additional binary abnormalityidentifiers indicating whether each of the measurements are normal orabnormal measurements, for example, based on whether or not thecorresponding measurement falls within predefined threshold boundaries.

The diagnosis data can include abnormality data 4442 for one or moreabnormalities, such as one or more abnormal rhythms. This can includedata localizing detected abnormalities in the raw signal data and/ordata characterizing one or more abnormalities, for example, based onmeasurement data 4443. The abnormality data 4442 can indicate the typeof abnormality present in the ECG, and can indicate any abnormal type ofrhythm or other abnormal features determined to be present in the ECG.

The diagnosis data 4440 can include one or more of medical codes 4447,which can include the same or different type of standardized medicalcodes as medical codes 4447, for example, based on the abnormality data4442. The diagnosis data 4440 can include annotation data 4448, forexample, utilized to provide text or visual indications of particularfeatures in the ECG waveform that are normal or abnormal, for examplefor display via a user interface in conjunction with display of the ECGwaveforms themselves to present some or all of the diagnosis data, forexample, based on the abnormality data 4442. Some or all of theannotation data can indicate some or all of measurement data 4443. Theannotation data 4448 can include some or all of the features ofannotation data 448. The diagnosis data 4440 can include report data4449, which can include natural language text describing the diagnosisand/or can include a formatted file that includes the captured image ECGprintout and/or one or more reconstructed pseudo-raw signals for one ormore leads. Report data 4449 can include some or all of the features ofthe report data 449.

The ECG entry can include diagnosis author data 4450, which can be thesame or different from annotation author data 450. The diagnosis authordata can indicate at least one diagnosis author identifier 4451, whichcan indicate function identifier in the medical scan analysis functiondatabase that generated some or all of the diagnosis data, such as anidentifier of the ECG processing function 4005. Alternatively or inaddition, the diagnosis author identifier 4451 can indicate a useridentifier in the indicating a user that generated some or all of thediagnosis data, such as a user that generated additional diagnosis dataas discussed in FIGS. 12F and/or a user that generated diagnosis datafor ECG entries utilized to train the ECG processing function 4005.

The ECG entry can include confidence score data 4460, which can bedetermined the same or different fashion as confidence score 460. Theconfidence score 460 can be determined as output of the ECG processingfunction 4005 in generating the diagnosis data, for example, indicatingone or more probabilities that one or more features indicated in thediagnosis data is correct and/or indeed present and/or one indicating ormore confidence intervals for one or more measurements or other featuresof the diagnosis data. The confidence score data can be further based ona determined quality of the captured image of the ECG printout, forexample, based on resolution, brightness, or other factors that affectimage quality. Diagnosis data generated for ECG printouts with lowerimage quality can be assigned lower confidence scores. The confidencescore data can be further based on an accuracy of one or more functionsutilized in performing the ECG processing function 4005. The confidencescore data can be further based on whether or not human review of theautomatically generated diagnosis data was provided, for example, wherethe confidence score data increases if human review confirms theautomatically generated diagnosis data. The confidence score data can beassigned a higher and/or highest score when the diagnosis data wasgenerated via human review and/or if the diagnosis data was utilized inone or more training sets utilized to train one or more inferencefunctions.

The ECG entry can include display parameter data 4470, which can includeinteractive interface feature data 4471, which can include some or allof the interactive interface feature data 471 and/or can indicateinteractive interface features utilized in displaying the diagnosis dataand/or corresponding ECG signal data via user interface 4075 and/orindicating layout and/or formatting preference for generating aformatted report such as report data 4449. The display parameter data4470 can indicate lead subset data 4472, indicating which leads thatwill have corresponding waveforms displayed via user interface 4075and/or can indicate a layout and/or an ordering of waveforms for displayvia user interface 4075 based on their corresponding leads as leadordering data 4473. The display parameter data 4470 can include waveformcropping data 4474 indicating whether and how one or more waveforms iscropped for display via user interface 4075. Some or all of the displayparameter data 4470 can be based on a user profile and/or user accountassociated with a user that utilizes the ECG interpretation system 4002and/or associated with a user that is associated with a correspondingclient device utilized to display the user interface 4075 on acorresponding display device.

The ECG entry can include training set data 4490, which can indicate oneor more training sets in which the ECG was utilized as training data,for example, to train one or more inference functions utilized by theECG processing function 4005 and/or to train one or more functions withparameters stored in the medical scan analysis function database 346.This can include one or more training set identifier data 4491, whichcan be the same or different from training set identifier data 491.

FIGS. 12C-12E illustrate steps in performing the ECG processing function4005. Some or all steps of the ECG processing function 4005 can utilizea same or different inference function that utilizes a computer visionmodel, a neural network, and/or or other machine learning model. Some orall of the functions illustrated in FIG. 12C can be trained by the ECGprocessing function 4005 or another subsystem 101 on training sets ofcaptured images of ECG printouts, on raw signal data of ECGs, and/orother ECG waveform data. Some or all of the functions illustrated inFIG. 12C can utilize any of the fields of corresponding ECG entries 4452for the set of training data as input labels and/or output labels totrain the corresponding model. Some or all of the functions illustratedin FIG. 12C can have function parameters set by an administrator viainteraction with a user interface. Some or all of the functionsillustrated in FIG. 12C can have a corresponding entry in the medicalscan analysis function database and/or can be trained and/or utilized byone or more other subsystems 101 as a medical scan analysis function asdiscussed herein.

As illustrated in FIG. 12C, a waveform detection function 4020 can beutilized to detect locations of waveforms in a captured image 4010,which can correspond to captured image 4417. The waveform detectionfunction 4020 can utilize a computer vision model trained on a pluralityof captured images of ECG printouts and/or trained on a plurality ofcaptured images of single waveforms included in ECG printouts to detectwaveforms in the captured image. The computer vision model cancorrespond to a neural network and/or a different machine learningmodel. The waveform detection function 4020 can utilize known layoutsand/or formatting features of ECG printouts to detect the plurality ofwaveforms. The known layouts and/or formatting features of ECG printoutscan further be utilized to detect the boundaries of the ECG printoutitself in the captured image, for example, where other backgroundoutside the detected boundaries of the ECG printout itself are ignoredand/or removed. Pre-processing can be performed to generate a grayscaleimage, to increase the contrast, to align a crooked image vertically,and/or to crop the ECG printout to a particular size, for example, basedon detecting the gridlines, based on aligning the position and/or sizeof the gridlines to preset position and/or scale, and/or based onincreasing contrast of the waveform and/or gridlines against otherportions of the captured image.

The waveform detection function 4020 can utilize the uniform grid of astandard ECG printout and/or uniform relative layout of waveformscorresponding to each of the twelve leads to determine locations of thewaveforms. Alternatively or in addition, the waveform detection function4020 can detect labels for some or all of the twelve leads, such as“aVR” or “V1” in the captured image. Alternatively or in addition, thewaveform detection function 4020 can detect known waveform shapefeatures in the captured image, and can differentiate between waveformsof different leads based on known differences in the corresponding knownwaveform shape features. Alternatively or in addition, the waveformdetection function 4020 can utilize uniform header formatting and/ortext detection to locate header data located at the top of the ECGprintout, and can determine the relative location of the waveforms basedon their known relative position from the header. Detection of theheader and/or known header formatting can be utilized to extractadditional information, such as some or all of additional details 4419,from the header of the printout. This can include patient name, age,hospital information, and/or some or all of measurement data 4443 and/orinitial diagnosis data utilized to generate diagnosis data.

Performance of the waveform detection function 4020 can result inwaveform location data 1-K, where K is equal to twelve or a differentnumber of waveforms in the image. This can, for example, indicate pixelcoordinates for borders that include each waveforms, for exampleindicated in FIG. 12D as borders 4013. While FIG. 12D indicates that theextended waveform of lead II is not utilized, in some embodiments, theextended waveform of lead II is utilized instead of or in addition tothe other waveform of lead II. While FIG. 12D depicts that the borders4013 are generated for all twelve leads, one or more leads can beignored for example if they are not included in the captured image, aredetermined to be obscured in the captured image, and/or if they are notutilized by further steps of the ECG processing function 4005 togenerate diagnosis data.

The image partitioning function 4022 can then be utilized based on thewaveform location data to generate waveform images 1-K. For example, asindicated in FIG. 12D, twelve waveform images 4014 can be generated bycropping the captured image in accordance with the borders 4013. Theindividual waveform images 1-K can be processed to generate pseudo-rawECG signal data 1-K by performing signal reconstruction function 4024.This can include detecting the waveform in the waveform image, forexample, in relation to detected gridlines of the ECG printout and/or bydetecting darker pixels for the waveform and the gridlines against thelighter pixels background of the ECG printout. For example, pseudo-rawECG signal data 4016, as illustrated in FIG. 12E, is generated as outputto performing the signal reconstruction function 4024. This can includegenerating a discrete plurality of coordinate pairs, each indicating atime based on a determined x-value of a point on the waveform and avoltage based on a corresponding determined y-value of the point of thewaveform based on the detected location of portions of the waveform inthe waveform image relative to the gridlines. An origin x and/or ygridline can be automatically detected based on formatting features ofthe ECG printout and/or based on the location of the waveform relativeto the plurality of gridlines and/or based on known relative boundariesof where the origin x and y gridlines should lie relative to features ofthe waveform.

As illustrated in FIG. 12E, signal reconstruction function 4024 caninclude a plurality of signal reconstruction functions specific to eachof the twelve leads, where each of the plurality of signalreconstruction functions is performed on the corresponding one of thetwelve waveform images to generate the corresponding pseudo-raw ECGsignal data. Some or all of these signal reconstruction functions can bethe same or different. Alternatively or in addition, performing thesignal reconstruction function 4024 and/or performing one or moreindividual, lead-specific signal reconstruction functions can includegenerating some or all of the measurement data 4443 to characterizesignificant features determined in the waveform images.

The signal reconstruction function 4024 can utilize known time intervalsand voltage intervals between gridlines along the vertical andhorizontal axes to generate some or all of the measurement data. Some orall of the measurement data can be detected directly from the waveformimage and/or from the pseudo-raw ECG signal data. Detecting the P, Q, R,S, and/or T wave can be based on utilizing a computer vision modeltrained on a training set of waveform images, a training set of raw ECGdata, and/or a training set of pseudo-raw ECG signal data as inputlabels, and a corresponding plurality of P, Q, R, S, and/or T wavecoordinate identifiers and/or measurements as output labels. Forexample, the output labels can indicate one or more coordinates of theECG signal data that correspond to the corresponding wave, and/or canindicate one or more pixel coordinates of the waveform image thatcorrespond to the corresponding wave. Signal reconstruction functionsfor different leads can be trained on separate training datacorresponding to waveform images, a training set of raw ECG data, and/ora training set of pseudo-raw ECG signal data for the corresponding lead.

In some embodiments, the shape of the QRS complex and/or one or moreindividual waves are detected in the waveform image data to determinethe location of the QRS complex and/or one or more individual waves inthe waveform image data, for example, by utilizing a computer visionmodel trained to detect shapes of the QRS complex and/or one or moreindividual waves. In some embodiments, a smoothing function is appliedto the pseudo-raw ECG signal data to remove extraneous noise and/orother minor waves that may be confused for the peaks of the P, Q, R, S,and/or T wave. In some embodiments, maximum amplitudes of the waveformimage and/or pseudo-raw ECG signal data are determined to detect thepeak of the corresponding wave. Once the waves are detected,corresponding interval measurements and heart rate measurements can bedetermined by utilizing the gridlines between wave locations and/or byutilizing corresponding time values of coordinates determined for thewaves.

The pseudo-raw ECG signal data 1-K and/or some or all measurement data4443 can be utilized as input to diagnosing function 4026. Alternativelyor in addition, the waveform images themselves can be utilized as inputto diagnosing function 4026, for example, where performance of thesignal reconstruction function 4024 is skipped. The diagnosing function4026 can utilize a plurality of diagnosing functions corresponding toeach lead, for example, where each of twelve outputs of the signalreconstruction function 4024 are utilized as input to a correspondingone of twelve diagnosing functions. Each of the twelve diagnosingfunctions can utilize models trained on their own set of training datacorresponding to waveform images, a training set of raw ECG data, and/ora training set of pseudo-raw ECG signal data for the corresponding lead.Diagnosis data can be determined for each lead separately, and a finaldiagnosis function can be utilized to consolidate the diagnosis data foreach lead to determine the diagnosis data. Alternatively, all twelveleads can be fed into one model as input, where the model was trained ona training data corresponding to waveform images, a training set of rawECG data, and/or a training set of pseudo-raw ECG signal data for alltwelve leads. In all cases, known diagnosis data for the training setcan be utilized as output data. The diagnosing function can utilize oneor more neural networks to generate the diagnosis data from thepseudo-raw ECG signal data, measurement data, and/or waveform images.The diagnosis data generated by performing diagnosing function 4026 caninclude some or all of diagnosis data 4440, can include automaticallygenerated treatment instructions based on the determined diagnosis,and/or can include prescription data such as an automatically generatedprescription and/or recommended prescription based on the determineddiagnosis.

Performing diagnosing function 4026 can utilize other information, suchas additional information 4419, ECG classifier data 4420, and/or patienthistory data 4430, for example, by utilizing models trained on thisother type of information as input labels. In some embodiments, otherECGs and/or other medical scans of the same patient are received aslongitudinal data, and are also processed as discussed herein togenerate some or all of the diagnosis data by utilizing the same ordifferent model trained on corresponding types medical scans asdiscussed herein, and/or by utilizing a single model trained to processboth the ECGs and additional medical scans as input to generatediagnosis data. Some or all of this additional information can beretrieved from a patient database corresponding to the patient, can beretrieved from the ECG database 4342, can be retrieved from a useraccount associated with the user that sent the captured image, and/orcan be retrieved from the client device in the same or differenttransmission as the captured image.

In some embodiments, some or all of the diagnosis data is generatedbased on the measurement data itself, for example, based on determiningone or more abnormal measurements and/or based on known rules fordiagnosing ECGs based on measurement data 4443. For example, thediagnosis data can be a deterministic function of the measurement data4443, and/or a partial diagnosis generated by utilizing the measurementdata 4443 can supplement and/or be utilized to confirm or deny otherdiagnosis data generated by applying diagnosing functions on waveformimages and/or pseudo-raw signal data.

In various embodiments, an ECG interpretation system 4002 includes atleast one processor and a memory that stores operational instructionsthat, when executed by the at least one processor, cause the ECGinterpretation system to receive, via a receiver, a captured image of anECG printout. A waveform detection function is performed on the capturedimage to determine a plurality of locations of a plurality of ECGwaveforms in the captured image. A plurality of waveform images aregenerated by partitioning the captured image based on the plurality oflocations, where each of the plurality of waveform images includes oneof the plurality of ECG waveforms. A plurality of pseudo-raw ECG signaldata is generated by performing a signal reconstruction function on eachof the plurality of waveform images, where each of the plurality ofpseudo-raw ECG signal data corresponds to one of the plurality ofwaveform images. Diagnosis data is generated by performing a diagnosingfunction on the plurality of pseudo-raw ECG signal data. The diagnosisdata is transmitted, via a transmitter, to a client device for displayvia a display device.

As discussed thus far, the ECG interpretation system 4002 can beutilized as an end to end automated service, where the diagnosis datagenerated automatically by the ECG interpretation system 4002 istransmitted back to the original client device from which the capturedimage was received. Similarly, the diagnosis data can be made availablefor access via a user account associated with the user that transmittedthe captured image, a user account associated with the patient for whomthe ECG was recorded, and/or a user account associated with a doctor ormedical entity providing care for the patient for whom the ECG wasrecorded. In this fashion, the diagnosis data can be accessed by anyclient device 120 where a user is logged in to their user account. Forexample, the client device 120 can store application data associatedwith the ECG interpretation system 4002 to execute an applicationallowing the user to access their diagnosis information when logged intotheir user account. Alternatively, a third party messaging service, suchas email, an SMS text-based messaging service, and/or internet-basedmessaging service can be utilized, where the captured image is receivedby the ECG interpretation system 4002 from an email address, phonenumber, or other handle associated with a user account, and where thediagnosis data is sent transmitted by utilizing the same messagingservice to the same email address, phone number, or other handleassociated with a user account.

ECG interpretation system 4002 can store, access, and/or maintain adatabase of user accounts that include corresponding user informationand/or corresponding ECG printouts. The user account information can bestored in ECG database 4342, for example, as patient identifier data4431 and/or retrieving entity identifier 4418. Alternatively or inaddition, patient identifier data 4431 and/or retrieving entityidentifier 4418 can include a unique identifier for a user account entryin the database of user accounts of the ECG interpretation system 4002.The user account can include a user login information and/or canindicate an email address and/or third party messaging service handle,identifying a user from whom a captured image is received via login to auser account, via an email from an email address identifying the user,and/or via a message via a third party messaging service via a handleidentifying the user. Similarly, the user account can indicate to whomdiagnosis data should be sent, and can be utilized to determine withuser account the diagnoses data should be made available to via userlogin to the user account, to determine which email address an emailthat includes the diagnosis data should be sent to via email, and/or todetermine which handle a message that includes the diagnosis data shouldbe sent via a third party messaging service. The user account can storecurrent and/or historical medical information for each user, userpreference data for use of user interface 4077 for each user and/or forgeneration of formatted medical reports, captured images andcorresponding diagnosis data generated for each user's use of thesystem, and/or other information for each user determined viainteraction with user interface 4077. The user account can be utilizedto generate one or more fields of an ECG entry 4452, and/or an ECG entry4452 can be utilized to populate one or more fields of the user account.

In some embodiments, human review is determined to be necessary ingenerating the diagnosis data. For example, the ECG interpretationsystem 4002 can determine diagnosis data generated for a captured imagerequires human review based on corresponding confidence score data 4460indicating or more probability values generated in generating thediagnosis data comparing unfavorably to one or more correspondingprobability thresholds, in response to a random quality assuranceprocess, in response to a request from a client device for a secondopinion via user interface 4075, and/or in response to the ECGinterpretation system otherwise determining human review is necessary.In some embodiments, all ECGs indicated as abnormal in the diagnosisdata are designated for human review. In some embodiments, all diagnosisdata corresponding to a rare condition that compares favorably to ararity threshold, a time-sensitive condition that compares favorably toa time-sensitivity threshold, and/or a severe condition that comparesfavorably to a severity threshold, are designated for human review. Insome embodiments, human review is never determined to be necessary, andthe diagnosis data generated automatically by the ECG interpretationsystem 4002 is utilized.

As illustrated in FIG. 12F, the diagnosis data generated by performingthe ECG processing function 4005 can correspond to partial and/orunconfirmed diagnosis data. The captured image of the ECG printout,reconstructed pseudo-raw signal data generated by performing the ECGprocessing function 4005, and/or the diagnosis data generated byperforming the ECG processing function 4005 can be transmitted to aselected, second client device, for example, that is different from theclient device from which the captured image was received. For example,the client device can be associated with a user in the user database,can correspond to a medical professional qualified to interpret ECGs,can correspond to an expert user in the user database, can correspond toan administrator of the ECG interpretation system 4002, and/or cancorrespond to a physician providing care for the patient for whom theECG was recorded.

This client device can display some or all of this information via theuser interface 4076. User interface 4076 can utilize features of userinterface 4075, such as those discussed in conjunction with FIGS. 13Dand/or 13E with regards to display of diagnosis data. User interface4076 can utilize features discussed in conjunction with the medical scanassisted review system 102 and/or the medical scan annotating system 106to prompt the user to review the ECG and/or the diagnosis data generatedby the ECG interpretation system 4002, and to provide annotations,corrections, and/or additional information via interaction with the userinterface 4076. The user can confirm the diagnosis, can provide edits tothe diagnosis, and/or can further characterize the ECG to provideadditional details. For example, if the diagnosis data generated by theECG interpretation system 4002 only indicates that the ECG is abnormaland/or that the ECG includes at least one abnormal measurement, thisdiagnosis data can be sent to the user, who can review the ECG andprovide additional diagnosis information, for example, indicating thatthe ECG corresponds to an atrial fibrillation. The user can interactwith the user interface to indicate particular portions of the waveformthat are abnormal and/or were utilized to determine the abnormalityexists as annotation data to be included in the additional diagnosisdata. The user can provide treatment steps and/or provide prescriptionmedication details, for example, if the user corresponds to a physicianof the user.

The user can indicate problems and/or corrections to the pseudo-rawsignal data and/or any measurement data 4443. The user can determinethat one or more models utilized by the ECG processing function 4005 beretrained and/or entered into remediation in response to detectingproblems with the diagnosis data. In some embodiments, in response to acorrection to the measurement data 4443, one or more coordinate pairs ofthe pseudo-raw signal data is automatically corrected to reflect themeasurement correction to change intervals, amplitudes, and/or locationsof one or more waves, and the graphical representation of updatedpseudo-raw signal data is displayed via user interface 4076. Similarlythe user can interact with the graphical representation of thepseudo-raw signal data to change one or more of the coordinate pairsbased on clicking and dragging the coordinate to its desired position,where the shape of the waveform of the graphical representation of thepseudo-raw signal data changes accordingly to display updated pseudo-rawsignal data, and/or where one or more measurements of measurement data4443 is updated to reflect the changes to the one or more coordinatepairs. The pseudo-raw signal data can be overlaid upon correspondingwaveform image 4014 as discussed in conjunction with FIG. 13E to aid theuser in editing the positions of the coordinates making up the waveformof the pseudo-raw signal data to better characterize the waveformdepicted in the captured image of the ECG printout.

The additional diagnosis data generated by the client device can betransmitted back to the ECG interpretation system 4002, and/or can betransmitted to the ECG database or other entity. A consolidationfunction 4006 can be performed on the additional diagnosis data and thepartial diagnosis data to generate the final diagnosis. This can includereplacing or supplementing the some or all of the partial diagnosis datawith some or all of the additional diagnosis data to generate the finaldiagnosis data. The final diagnosis data to be sent back to the originalclient device from which the captured image was received.

FIGS. 13A-13E illustrate example embodiments of the user interface 4075presented to the user in conjunction with use of a client devicecommunicating with the ECG interpretation system 4002. The userinterface 4075 can be displayed in conjunction with execution of a webapplication executed in a web browser displayed by a display device ofthe client device and/or in conjunction with execution of locally storedapplication data such as a mobile application associated with the ECGinterpretation system 4002. The application can allow and/or requirethat the user create and/or login to a user account to interact withsome or all of the features presented in FIGS. 13A-13E.

The client device and/or user account can correspond to the patient ofwhom the ECG printout was recorded, and/or can correspond to a medicalprofessional or any other person providing care for the patient. Forexample, the patient can take or upload a picture and/or scan of theirown ECG printout they received at a visit to a doctor's office orhospital and/or a picture taken of the display on the ECG monitor whilethe ECG monitor was recording their electrical activity. The patient canreceive their diagnosis data from the ECG interpretation system 4002 inresponse and/or can designate a physician and/or other entity to receivethe diagnosis data.

Alternatively, a physician or other medical professional can take and/orupload a picture and/or scan of the ECG printout with their own clientdevice, can receive their diagnosis data from the ECG interpretationsystem 4002 in response and/or can designate that the diagnosis data besent to a different entity. For example, a medical professionalproviding care for a patient can take or upload a picture and/or scan ofthe patient's ECG printout, a picture taken of the display on the ECGmonitor, and/or a screenshot of the display on the ECG monitor. Inparticular, the doctor can utilize the ECG interpretation system 4002 ifthey are not adept at interpreting ECGs and/or if they encounter an ECGof a patient that is odd and/or difficult to interpret.

As another example, the client device can be associated with a firstresponse medical professional providing emergency care for a patientoutside of a normal hospital setting, for example, at the place where apatient undergoing an emergency is found and/or in an emergencytransport vehicle such as an ambulance and/or helicopter. The firstresponse medical professional can utilize the client device to take apicture of the display on an ECG monitor while the corresponding ECGmachine is recording to the patient for automatic interpretation by theECG interpretation system 4002. Alternatively the client device canconnect to the ECG machine to display ECG readings recorded by the ECGmachine on its own display device, and can automatically utilize the rawECG readings and/or a screenshot of the ECG readings displayed on thedisplay device of the client device as the captured image. The clientdevice can be associated with the first response medical professionaland/or the emergency transport vehicle can be equipped with the clientdevice. The diagnosis data can be transmitted back the client device fordisplay to the first response medical professional, for example, wherethe diagnosis data includes emergency care instructions to the firstresponse medical professional based on abnormalities detected in the ECGwaveforms. Alternatively or in addition, the diagnosis data can betransmitted to a database and/or client device associated with ahospital or other medical entity where the first response medicalprofessional is taking the patient. This can allow medical professionalsat the hospital to view the diagnosis data before the patient arrives atthe hospital, enabling the medical professionals to prepare for thepatients arrival, and/or ensuring they are ready to provide immediatecare to the patient once they arrive based on results and/or treatmentinstructions included in the diagnosis data.

While FIGS. 13A-13E depict the presentation of the user interface via adisplay device of a mobile device, any display device associated withany client device, such as a personal computer, laptop, tablet, or otherclient device can display the user interface 4075. Features of theexample user interface embodiments presented in FIGS. 13A-13E can bepresented in any configuration, and other embodiments can includeadditional features instead of or in addition to some or all of thefeatures presented in FIGS. 13A-13E to perform the same or differentfunctions as discussed in conjunction with FIGS. 13A-13E.

FIG. 13A presents an example embodiment of a user interface 4075presented to the user to prompt the user to generate and/or upload thecaptured image of the ECG printout. The user can elect to upload anexisting captured image of the ECG printout from memory of the clientdevice. The user can elect to use an image captured device 4012 coupledto the client device to generate the captured image. For example, ifthis option is selected, the client device can automatically sendinstructions to, or otherwise communicate with, the image capture device4012, such as a camera coupled to the client device and/or a scannercommunicating with the client device, to cause the image captured device4012 to capture the image data. Once the captured image is uploadedand/or generated, the captured image can be transmitted to by the clientdevice to the ECG interpretation system 4002 and/or to the ECG database4342 for retrieval by the ECG interpretation system 4002.

In some embodiments, the client device 120 includes health monitoringcapabilities that can measure heart rate, hearth rhythm, and/or otherhealth data. For example, the client device 120 can correspond to asmart watch or other wearable device worn by the user that collects thehealth data, or can communicate with and/or receive data from a wearabledevice worn by the user that collects the health data. As anotherexample, the client device 120 can communicate with or upload data froma user's pace maker device to retrieve ECG data, abnormal rhythm data,and/or other health data collected by the pace maker. The client device120 can communicate with or upload data from other medical devices, forexample, that specializing in capturing ECG data and/or heart rhythmdata. This captured ECG data, heart rhythm data, and/or other healthdata can be utilized instead of or in addition to the captured image ofan ECG printout, for example, as ECG data and/or patient history data4430 utilized as input to the ECG processing function 4005 to generatethe diagnosis. The user can elect to upload and/or automaticallyretrieve some or all of this health data via the user interface 4075instead of or in addition to the captured image. This health data can betransmitted to by the client device to the ECG interpretation system4002 and/or to the ECG database 4342 for retrieval by the ECGinterpretation system 4002 instead of or in addition to the capturedimage, where the ECG processing function 4005 is performed only on thishealth data to generate the diagnosis data and/or where the ECGprocessing function 4005 is performed on this health data as patienthistory data 4430 in conjunction with the captured image to generate thediagnosis data.

FIG. 13B illustrates an embodiment where the user elected to utilize theimage captured device 4012 to generate the captured image. For example,a camera of a mobile device implemented as the client device isautomatically utilized to allow the user to take a picture of the ECGprintout. The user can view the captured image and can elect to retakethe image if needed and/or can elect to send the image as presented. Insome embodiments, the client device can automatically detect whether theentirety of the ECG printout is included in the frame and/or canautomatically detect whether the resolution and/or image quality of theECG printout is satisfactory, for example, based on detecting thepresence of the gridlines, based on detecting the presence of therequired number of waveforms, and/or based on detecting the presence ofall four borders of the printout within the image. In such embodiments,the user will not be permitted to take the picture and/or the user willbe required to retake the picture if these criteria are determined notto be met.

FIG. 13C illustrates an embodiment of the user interface 4075 where theuser can enter additional information to supplement the captured imageof the ECG printout. For example, this can be entered in conjunctionwith the user creating information for their user account and/or theuser can be prompted to enter additional information in response togenerating and/or uploading the captured image as discussed inconjunction with FIGS. 13A and 13B. The user can enter health history,current medications, and/or current symptoms manually, for example, byanswering a series of health history questions presented by the userinterface 4075, and/or can elect to upload health history document,prescription documents, and/or other documents, where the client deviceextracts the relevant health history information from the documentautomatically. Past medical reports, such as report data 449 and/or4449, can be uploaded by the user and/or can be automatically retrievedby the client device 120 and/or the ECG interpretation system 4002 froma medical scan database 342, ECG database 4342, and/or a patientdatabase. Similarly, other data relevant to the patient stored by amedical entity, such as medical scans or past ECGs, or other entries ofa medical scan entry 352 and/or ECG entry can be retrieved from medicalrecords of the patient. For example, the user can enter information fortheir hospital and/or physician and/or can give permission to theirhospital and/or physician to release their medical information to theECG interpretation system 4002. Some or all fields of health history,current medications, current symptoms, and/or medical reports can bemandatory and/or optional. In some embodiments, none of this informationis required, but any of this information provided by the user istransmitted to the by the client device.

The entered information by utilizing user interface 4075 of FIG. 13Cand/or any subsequent prompts via user interface 4075 is sent by theclient device 120 to the ECG interpretation system 4002 and/or to theECG database 4342 for retrieval by the ECG interpretation system 4002.For example, this information can be utilized as patient history data4430 for use as input to the ECG processing function 4005 in conjunctionwith the captured image to generate some or all of the diagnosis data.In particular, a patient's health history, current medications, currentsymptoms, or other medical documents can provide insight to diagnose thepatient and/or to determine a cause of abnormal rhythms detected in thediagnosis data generated by performing ECG processing function 4005. Forexample, the diagnosis data can include specialized treatment data,prevention instructions, and/or prescription data for any abnormalitiesdetected in the ECG determined based on this additional information ofpatient history data 4430.

FIGS. 13D and 13E present embodiments of displaying the diagnosis datareceived from the ECG interpretation system 4002 via user interface 4075once the ECG printout and/or any additional information provided by theuser is processed. FIG. 13D illustrates diagnosis data displayed astext, indicating a diagnosis of premature ventricular contractions. Thediagnosis data can be presented to include any additional informationpresented in the diagnosis data such as treatment options, prescriptiondata, information about how this diagnosis was detected in the ECG,confidence score data indicating probabilities that this diagnosis isaccurate, pseudo-raw signal data and/or measurement data 4443automatically. In some embodiments, the user can elect to view some orall of this additional information via selecting one or morecorresponding options presented via the user interface 4075.

The user can elect to generate and/or view a formatted report forexample as a PDF or other file format, where the report is generated bythe client device 120 by utilizing the diagnosis data. Alternatively,the formatted report can be generated by and received from the ECGinterpretation system 4002 in the diagnosis data automatically. Theformatted report can utilize formatting preferences and/or otherpreferences to generate the formatted report, for example, indicated inthe user account for the user and/or specified via other input to one ormore other prompts presented via user interface 4075. The formattedreport can correspond to report data 4449, and can include a processedor unprocessed captured image of the ECG printout; can include some orall of measurement data 4443; can include a graphical display ofpseudo-raw signal data reconstructed by the ECG interpretation system4002 for one or more leads, for example as shown in FIG. 12E aspseudo-raw ECG signal data 4016; can include annotations such asannotation data 4448 overlaid upon the to the pseudo-raw signal data orwaveforms of the ECG printout itself, indicating abnormal rhythms,indicating measurement data 4443 and/or the P, Q, R, S and/or T wave,and/or indicating other significant features detected in the waveform ofone or more leads; can include treatment data and/or prescriptionmedication to be taken to mitigate the effects of the diagnosis; and/orcan indicate any other diagnosis data 4440 generated by performing theECG processing function 4005 and/or indicating other fields of acorresponding ECG entry 4452. Some or all of this information of theformatted report can be displayed in the same or different format viathe interactive interface 4075.

The user can elect to view the reconstructed signals, as discussed infurther detail in FIG. 13E. The user can elect to view theauto-generated measurements, where some or all of measurement data isdisplayed as text and/or is visually indicated on the waveforms of thepseudo-raw signal data and/or on the original and/or processed capturedimage of the ECG printout.

The user can elect to send their results their physician or othermedical entity automatically, for example, based on the contactinformation or an account associated with the physician. Thisinformation can be indicated in the user's account or can otherwise beentered by the user, for example, where the user indicates an emailaddress or other messaging handle for the physician via interaction withthe user interface 4075 and where some or all of the diagnosis data istransmitted in a message via email or other messaging service to thephysician accordingly. In some embodiments, the physician can login totheir own user account and can view diagnosis data of one or more oftheir patients in a similar fashion as discussed in conjunction withFIGS. 13D and 13E via their own interactive interface 4075 and/or 4076.In some embodiments, the diagnosis data is automatically transmitted toa client device associated with the physician by the ECG interpretationsystem 4002 once it is generated based on physician information for thepatient in their user account.

The user can elect to send a prescription prescribed based on thediagnosis data to a pharmacy immediate and/or as soon as it isavailable. their physician In some embodiments, a prescription isgenerated automatically in the diagnosis data generated by performingECG processing function 4005 and/or is received from a physician thatreviewed the diagnosis data. For example, the prescription can begenerated via interaction with user interface 4076 in generating theadditional diagnosis data as discussed in conjunction with FIG. 12F.Alternatively or in addition, the prescription can be generated by aclient device associated with the user's physician in response to theuser electing to send the diagnosis data to their physician. Thediagnosis data can include one or more recommended prescriptions and thephysician interacting with user interface 4076 and/or the user'sphysician that received the diagnosis data can select from the one ormore recommended prescriptions in the diagnosis data. The user accountcan include pharmacy information and/or the user can enter pharmacyinformation via the user interface 4075. The prescription can beautomatically sent to the pharmacy indicated in the user account once itbecomes available.

The user can elect to request a second opinion, for example, in responseto determining the results in the diagnosis data is unconvincing,incomplete, and/or in response to determining additional review isnecessary based on the review of the diagnosis data. This can prompt thetransmission of the diagnosis data to the client device for additionalreview to receive edits and/or additional diagnosis data as illustratedin FIG. 12F. The user selected for the second opinion can beautomatically determined by the ECG interpretation system 4002 and/orcan be selected by the user.

The user can elect to review more information about their diagnosisand/or treatment of their diagnosis, for example, where the clientdevice displays medical articles and/or websites specific to a conditionindicated in their diagnosis data. Internet links to medical articlesand/or websites, and/or documents that include the articles oradditional information about the diagnosis can be selected by the ECGinterpretation system 4002 based on the diagnosis data for transmissionin conjunction with the diagnosis data, where the user is automaticallydisplayed the links and/or the documents via the interactive interface4075. The user can select a link to be routed to a corresponding articleand/or website for display to the user via a web browser.

FIG. 13E illustrates an example embodiments of the user interface 4075displayed in response to user selection to view the reconstructedsignals and/or displayed automatically. The reconstructed signals caninclude a graphical representation of the pseudo-raw signal datagenerated by performing the ECG processing function 4005. As shown inFIG. 13E, annotation data 4448 indicating diagnosed prematureventricular contractions or other waveform features characterizing thediagnosis can be visually indicated, and/or other features such as thenormal beats can be visually indicated. The measurement data and/orother annotation data can similarly can be indicated in the waveform,where corresponding portions of the waveform are highlighted, displayedin a different color, outlined, bolded, circled or surrounded by asegmented shape, or otherwise visually indicated.

The waveforms for some or all twelve leads can be displayed one at atime or all at the same time. For example, the user can elect to advanceto the next lead, and each lead can be displayed one at a time. Theordering of the leads and/or how many lead waveforms are displayed in asingle view can be set based on user preferences and can be stored asuser account data and/or the ordering of the leads can be preset.Alternatively, the first lead displayed and/or the ordering of leadsdisplayed can be determined automatically by the ECG interpretationsystem 4002 in performing the ECG processing function 4005, for example,where lead waveforms with most abnormal findings, lead waveforms withmost significant findings, and/or lead waveforms with most visuallyobvious findings are displayed first.

The user can elect to view the original captured image of a cropped,single waveform such as waveform image 4014 corresponding to thecurrently displayed waveform and/or can display the entirety of thecaptured image of the ECG printout in the same or different view as thereconstructed signal. Alternatively, some or all of the processed and/orunprocessed captured image is displayed automatically. The annotationdata 4448 can similarly be displayed as overlaid upon the waveformdetected in the waveform image 4014. In some embodiments, the pseudo-rawsignal data can be overlaid graphically on top of the correspondingwaveform image 4014, for example in a different color from the color ofthe waveform in the waveform image. Alternatively, the pseudo-raw signaldata and corresponding waveform image 4014 can be displayed in adjacentviews of the user interface.

The user can elect to view an example of one or more normal signals, forexample, corresponding to one or more leads currently being viewed. Thenormal signal can be the same normal signal for every user. In someembodiments, the normal signal can be generated by the ECGinterpretation system 4002, for example, by detecting the abnormalfeatures of the waveform in the pseudo-raw signal data and by removingthe abnormal features from waveform in the pseudo-raw signal data, forexample, where an inverse signal corresponding to the portions of thewaveform that are abnormal is added to the pseudo-raw signal data togenerate pseudo-normal signal data. The pseudo-normal signal data can bepresented in conjunction with the pseudo-raw signal data and/or thewaveform image 4014, for example, in an adjacent view. Alternatively,the pseudo-normal signal data can be overlaid graphically on top of thecorresponding waveform image 4014 and/or the pseudo-raw signal data, forexample, in a different color from the color of the waveform in thewaveform image and/or from the color of the graphical representation ofthe pseudo-raw signal data.

FIG. 14 presents a flowchart illustrating a method for execution by anECG interpretation system 4002 that stores executable instructions that,when executed by at least one processor, cause the ECG interpretationsystem 4002 to perform the steps below.

Step S002 includes receiving, via a receiver, a captured image of an ECGprintout. Step S004 includes performing a waveform detection function onthe captured image to determine a plurality of locations of a pluralityof ECG waveforms in the captured image. Step S006 includes generating aplurality of waveform images by partitioning the captured image based onthe plurality of locations, where each of the plurality of waveformimages includes one of the plurality of ECG waveforms. Step S008includes generating a plurality of pseudo-raw ECG signal data byperforming a signal reconstruction function on each of the plurality ofwaveform images, where each of the plurality of pseudo-raw ECG signaldata corresponds to one of the plurality of waveform images. Step S010includes generating diagnosis data by performing a diagnosing functionon the plurality of pseudo-raw ECG signal data. Step S012 includestransmitting, via a transmitter, the diagnosis data to a client devicefor display via a display device.

In various embodiments, the ECG printout was generated by an ECG machinebased on electrical activity recorded by the ECG machine, and thecaptured image was generated by a utilizing an image capture device togenerate the captured image from the ECG printout. In variousembodiments, the image capture device corresponds to a camera. Invarious embodiments, the camera is coupled to a mobile device, and thecaptured image is transmitted to the ECG interpretation system by themobile device. In various embodiments, the client device that receivesthe diagnosis data corresponds to the mobile device. In variousembodiments, the image capture device corresponds to a scanner.

In various embodiments, the waveform detection function utilizes acomputer vision model trained on a training set of waveform images. Invarious embodiments, the waveform detection function utilizes a computervision model trained on a training set of ECG printouts. In variousembodiments, the waveform detection function utilizes standardizedlayout information, where the standardized layout information indicatesrelative positions of waveforms in standard ECG printouts. In variousembodiments, the standardized layout information indicates relativelocations of exactly twelve waveforms. Each of the exactly twelvewaveforms corresponds to one of twelve ECG leads. The plurality ofwaveform images includes exactly twelve waveform images corresponding tothe twelve ECG leads. In various embodiments, the waveform detectionfunction utilizes standardized formatting feature information, where thestandardized formatting feature information indicates at least onerelative position of at least one additional feature in standard ECGprintouts. Performing the waveform detection function includes detectingthe at least one additional feature in the ECG printout.

In various embodiments, the signal reconstruction function utilizes acomputer vision model trained on a training set of waveform images. Invarious embodiments, the signal reconstruction function determines aplurality of discrete points for one of the plurality of waveformimages, where a corresponding one of the plurality of pseudo-raw ECGsignal data includes the plurality of discrete points. In variousembodiments, the diagnosing function is trained on a training set of rawECG signal data, where each raw ECG signal data in the training setincludes discrete points characterizing the raw ECG signal data. Invarious embodiments, the signal reconstruction function determines apseudo-raw waveform shape for one of the plurality of waveform images,where a corresponding one of the plurality of pseudo-raw ECG signal dataincludes the pseudo-raw waveform shape. In various embodiments, thediagnosing function is trained on a training set of raw waveform shapescorresponding to raw ECG signal data. In various embodiments, thetraining set of raw waveform shapes corresponds to image data thatincludes raw waveform shapes recorded by at least one ECG machine.

In various embodiments, the diagnosing function is trained on a trainingset of raw ECG signal data. Each raw ECG signal data in the training setincludes raw ECG signal data for each of twelve ECG leads, and theplurality of pseudo-raw ECG signal data includes twelve pseudo-raw ECGsignal data corresponding to the twelve ECG leads. The diagnosingfunction is performed on the twelve pseudo-raw ECG signal data as inputto generate the diagnosis data.

In various embodiments, performing the diagnosing function on theplurality of pseudo-raw ECG signal data includes generating twelvesub-diagnosis data, where each of the twelve sub-diagnosis data isgenerated by performing one of twelve inference functions on acorresponding one of twelve pseudo-raw ECG signal data. Each of thetwelve inference functions corresponds to one of twelve ECG leads and istrained on a training set of raw ECG signal data for the correspondingone of the twelve ECG leads. A final inference function is performed onthe twelve sub-diagnosis data to generate the diagnosis data.

In various embodiments, the captured image is received from the clientdevice, the ECG printout corresponds to a patient, and the client deviceis associated with the patient. In various embodiments, the clientdevice is associated with a medical professional responsible for thepatient. In various embodiments, the ECG printout is received in anfirst email transmission from a first email address associated with arequestor to a second email address associated with the ECGinterpretation system, and the diagnosis data is transmitted in a secondemail transmission from the second email address to the first emailaddress. In various embodiments, the requestor is a user of to theclient. In various embodiments, the second email transmission includes adocument for display to the requestor via the client device. Thedocument includes the diagnosis data and the plurality of pseudo-raw ECGsignal data. In various embodiments, the diagnosis data is displayed viathe display device in conjunction with the captured image. In variousembodiments, the diagnosis data is displayed via the display device inconjunction with the plurality of pseudo-raw ECG signal data via a userinterface. In various embodiments, the diagnosis data is displayed viathe display device in conjunction with the plurality of pseudo-raw ECGsignal data via a user interface.

In various embodiments, the diagnosis data indicates at least oneabnormality in at least one of the plurality of pseudo-raw ECG signaldata. The at least one abnormality is visually indicated in the displayof the at least one of the plurality of pseudo-raw ECG signal data. Invarious embodiments, normal pseudo-raw ECG signal data is generated byremoving the at least one abnormality from the at least one of theplurality of pseudo-raw ECG signal data. The normal pseudo-raw ECGsignal data is displayed via the display device in conjunction with thediagnosis data and the plurality of pseudo-raw ECG signal data.

In various embodiments, as discussed in further detail in conjunctionwith FIGS. 15A-15C, the method includes the following additional steps,which can be performed in addition to or instead of some or all of thesteps of FIG. 14. The additional steps include receiving, via thereceiver, a captured image of a patient's skin. The additional stepsfurther include generating second diagnosis data by performing a seconddiagnosing function on the captured image of the patient's skin, wherethe second diagnosing function utilizes a computer vision model trainedon a plurality of images of skin corresponding to a plurality of skinconditions. The additional steps further include transmitting, via thetransmitter, the second diagnosis data to the same or different clientdevice for display via a display device.

While FIGS. 12A-12F, FIGS. 13A-13E, and FIG. 14 illustrates embodimentsspecific to ECGs, some or all of the functionality of the ECGinterpretation system 4002 and/or the user interfaces 4075 and/or 4076displayed by client devices 120 can similarly be applied to any othercaptured images that include medical information. FIGS. 15A-15Cillustrate an embodiment of a captured image processing system 4004,which can utilize features of, be integrated within, and/or communicatewith the ECG interpretation system 4002 and/or one or more othersubsystems 101. Instead of or in addition to generating diagnosis databased on images of ECG printouts, diagnosis data can be generated by thecaptured image processing system 4004 based on captured images of othermedical information. As illustrated in FIG. 15A, a captured image can bereceived from a client device. The captured image processing system 4004can perform a processing function 4008 to generate diagnosis data forthe captured image, which can include some or all of the diagnosis data440 and/or diagnosis data 4440. The processing function 4008 can includeany of the medical scan analysis function and/or inference functiondiscussed herein, and/or can utilize a computer vision model trained onone or more types of captured images and corresponding diagnosis data.

In some embodiments, the captured image processing system 4004 firstperforms a image classifier function 4007, for example, to determine ananatomical region of the image, to determine anatomical features in theimage that need to be diagnosed, and/or to otherwise classify the imageinto one of a plurality of image types. Once the image type isdetermined by performing the image classifier function 4007 on thecaptured image, one of a plurality of different processing functions4008, each corresponding to one of the plurality of image types, can beselected for performance on the image data. In some embodiments, thecaptured image is pre-processed with filters to change brightness,contrast, orientation, and/or to crop the image before the imageclassifier function 4007 and/or processing functions 4008 is applied.

The diagnosis data generated by performing the processing functions 4008can be transmitted to the same and/or different client device fordisplay via an interactive interface 4077. The captured image and/ordiagnosis data can be transmitted to medical scan database 342. Forexample, the medical scan database can include entries for capturedimages of medical documents and/or captured images of the body taken bya camera as discussed herein, where image data 410 corresponds to one ormore captured still images and/or a captured video. Alternatively or inaddition the captured image and/or diagnosis data can be sent to ECGdatabase 4342 and/or a different database that stores captured images,diagnosis data, and/or some or all of the fields of entries of medicalscan database 342 and/or ECG database 4342. The captured image and/ordiagnosis data can be alternatively or additionally be transmitted toany other medical entity such as a doctor of the patient and/or toanother client device to generate additional diagnosis data as discussedin conjunction with FIG. 12F, for example, where user interface 4076 isutilized to generate additional diagnosis data corresponding tosupplemental information and/or corrections to the diagnosis data viahuman review, and where final diagnosis data is generated andtransmitted based on the additional diagnosis data.

In some embodiments, captured images received by the captured imageprocessing system 4004 correspond to captured images such as photos orscans of physical printouts of any of the medical scans discussedherein, captured images such as photos or scans of physical printouts ofany of the report data discussed herein, and/or captured images such asphotos or scans of any printouts for any medical information generatedby a machine and/or a medical professional. In such embodiments, thecaptured image can be processed by utilizing a computer vision modeland/or natural language model for example, as discussed in conjunctionwith the medical scan image analysis system 112, the medical scandiagnosing system 108, and/or the medical scan natural language analysissystem 114. The image classifier function 4007 and/or processingfunction 4008 can utilize any of the medical scan analysis functiondiscussed herein. The captured image can be pre-processed by utilizingadditional image processing to best capture and/or enhance the pertinentinformation of the scan and/or report in the printout. The imageclassifier function 4007 can determine one of a plurality of medicalscan types and/or anatomical regions by utilizing the input qualityassurance function and/or another computer vision model trained oncaptured images of medical scans with output labels indicating scantype, anatomical region, and/or other scan classifier data 420. Theprocessing function 4008 can correspond to a medical scan inferencefunction 1105 of the corresponding category, and/or can be trained on atraining set of captured images of physical printouts of medical scansof the corresponding category with known diagnosis data 440 as outputlabels. Alternatively, some or all of the medical scans in the trainingset can correspond to image data 410 of the original medical scans,rather than a captured image of a physical printout of the medical scan.

Alternatively or in addition, captured images of physical, anatomicalfeatures of a patient, taken by a camera, can be similarly be processedby the captured image processing system to generate diagnosis data. Forexample, a patient can utilize a camera on their mobile device to take apicture of a concerning abnormality on their skin and/or an externallyobservable abnormality that is otherwise visually observable via thehuman eye and/or via the image capture device 4012 communicating and/orcoupled to their client device. This captured image can be transmittedby the client device to the captured image processing system 4004 forprocessing. The patient and/or their dermatologist or other doctor canreceive and view diagnosis data generated by the captured imageprocessing system 4004 in response via the same or different clientdevice.

The image classifier function 4007 can detect the anatomical region,type of bodily tissue and/or bodily fluid captured, and/or otherclassifying information for the captured image to determine theprocessing function 4008. For example, the image classifier function canutilize a computer vision model trained to detect anatomical regions ofthe human body and/or types of bodily tissue and/or bodily fluid in thecaptured image, for example, where the computer vision model is trainedon a plurality of captured images of the human body with and/or withoutabnormalities included, where the output labeling data indicates theanatomical regions, type of bodily tissue, type of bodily fluid, and/orother classifying information in the captured image. Alternatively or inaddition, the image classifier function 4007 can be utilized todistinguish between captured images of the body by a camera and imagescorresponding to medical documents such as captured images of printoutof medical scans, ECG printouts, or medical reports.

Some or all processing functions 4008 can be trained on captured imagescorresponding to the classifying type. For example, one processingfunction can be trained to process pictures of the hand and a secondprocessing function can be trained to process pictures of the back. Asanother example, one processing function can be trained to processpictures of skin conditions to provide dermatological diagnosis, anotherprocessing function can be trained to process pictures of the inside ofthe mouth to provide a dental and/or orthodontal diagnosis, and/oranother processing function can be trained to process pictures of theeye to provide an ophthalmological diagnosis. As another example, oneprocessing function can be trained to diagnose pictures of bruised skin,another processing function can be trained to diagnose pictures of skinthat include acne, another processing function can be trained todiagnose pictures of skin with a rash, another processing function canbe trained to diagnose pictures of skin with a rash, another processingfunction can be trained to diagnose pictures of skin with a cuts and/orabrasions, and/or another processing function can be trained to diagnosepictures of skin with a burn. Some processing functions 4008 can betrained to generate RECIST data for moles and/or skin lesions incaptured images of the skin, for example, by utilizing one or moremeasurement functions of the lesion tracking system 3002 of FIGS.16A-16D. Each processing function 4008 can utilize a computer visionmodel trained to diagnose conditions of the corresponding type of image,for example, where each processing function 4008 is trained on atraining set of images of the corresponding type as input data and oncorresponding diagnosis data as output labels.

In some embodiments, multiple processing functions 4008 are performed inresponse to output of the image classifier function 4007 indicating thecaptured image falls under multiple categories. Alternatively or inaddition, multiple processing functions 4008 are performed in accordancewith a hierarchy of processing functions 4008. For example, a firstprocessing function 4008 trained to detect skin conditions is performedfirst on the captured image in response to determining the capturedimage is a picture of the skin. The diagnosis data generated by thefirst processing function can indicates a rash, and second processingfunctions 4008 trained to diagnose particular types of rashes can beselected to be performed next on the captured image to provide a moredetailed diagnosis. In some embodiments, a single processing function4008 is performed on all captured images, and/or no image classifierfunction 4007 is performed.

In some embodiments, the user can provide additional informationdescribing the condition. This additional information can correspond towhat part of the body the image includes, for example if the image isvery zoomed in; can include image measurement and/or other observationsof the abnormal condition taken by the user; can include symptoms suchas pain and/or itchiness of a skin condition and/or other symptoms; caninclude current medication being taken, medical history, family medicalhistory, and/or can include other information that can be entered viauser interface 4077 by the user for transmission to the captured imageprocessing system 4004 in conjunction with the captured image. Some orall of the additional information can correspond to information enteredby the user to user interface 4075 as discussed in conjunction with FIG.13C. This additional information can correspond to and/or can beutilized to determine patient history data 430 and/or one or more otherfields of a corresponding entry in the medical scan database. This canbe utilized by the image classifier function 4007 to determine whichprocessing functions 4008 are performed on the captured image. This canalso be utilized as input to one or more processing functions 4008 togenerate some or all of the diagnosis data, for example, where aprocessing function 4008 utilized similar types of additional data asinput labels in the training data to train the processing function 4008.

The captured image can correspond to a single, still, image, photographand/or other picture, for example, received as a PNG, JPEG, or otherimage format for a single image. In some embodiments, a set of multiplecaptured images can be received, for example, corresponding to multiplepictures taken of an abnormal condition appearing on the same ordifferent part of the body from the same or different angle. The same ordifferent processing function 4008 can be performed on each of the setof captured images, and diagnosis data generated for each of the set ofmultiple captured images can be used in conjunction to generate thediagnosis data.

In some embodiments, the set of multiple captured images corresponds totemporal data taken over time, where the captured images show a changein the condition and/or or lack in change in the condition over time.The processing function 4008 can generate diagnosis data based on adetected and/or measured change in the condition and/or in the amount oftime elapsed between captured images. For example, the processingfunction 4008 can utilize the lesion tracking system 3002 of FIGS.16A-16D to generate diagnosis data that includes lesion measurement dataand/or lesion tracking data. In particular, the processing function 4008can be performed on pictures of skin lesions and/or moles on skin overtime, where measurement change data is generated in generating thediagnosis data and/or where RECIST data is included in the diagnosisdata. In some embodiments, measurement data and/or measurement changedata is applied to pixel values and/or an established color in thecaptured image and/or multiple captured images taken over time todetermine color and to track the change in color over time.

The client device can transmit video data, for example, to capture theabnormal condition from the different angles and/or different parts ofthe body, where some or all frames of the video are processed as a setof multiple captured images. Alternatively, the abnormal condition cancorrespond to a motion that can be captured by video but not by a stillimage. For example, an abnormal twitch, shaking, reflexive movement,rate of flow of a bodily fluid, dilation of pupils or other eyemovement, and/or other bodily motion that can be captured via video canbe included in the video data. The motion can be tracked and/or measuredover the multiple frames in performing the processing function 4008 togenerate the diagnosis data. For example, the processing function 4008can be trained on training data that includes video data and/or includesa plurality or frames as input labels and includes correspondingdiagnosis data as output labels.

As another example, a first response medical processional can utilize acamera on their mobile device and/or other client device to take a totake one or more pictures of externally observable medical conditionssuch as a severe wound that requires emergency care, a severe allergicreaction that requires emergency care, and/or any other externallyobservable condition that can be captured via a camera and/or anotherand/or via the image capture device 4012 communicating and/or coupled tothe client device of the first response medical processional in a singleimage or a set of images. In some embodiments, the medical processionalcan utilize a camera on their mobile device and/or other client deviceto take a video to better capture the externally observable medicalconditions and/or to capture motion-based conditions such as a seizure,a person who is chocking, a rate of blood loss from a wound, eyedilation or eye movement, or other condition that is observable asmotion over multiple frames. In such embodiments, the image capturedevice 4012 can include a radiological medical imaging device or otherspecialized medical imaging equipment instead of or in addition to acamera. The client device can transmit the captured image to thecaptured image processing system 4004 for processing, and the diagnosisdata can be sent back to the first response medical processional inconjunction with emergency care instructions based on the diagnosis toaid the first response medical processional in providing care.Alternatively or in addition, the captured images and/or the diagnosisdata can be sent to the hospital and/or medical entity the patient isrouted to directly for view by a medical professional at the hospitalbefore the patient arrives, allowing the medical professional to providespecialized care based on the diagnosis data.

As another example, a bystander observing a person undergoing a medicalemergency can utilize a camera on their mobile device, or on the mobiledevice of the person, to take one or more picture and/or a video of thesevere, externally observable medical condition. The mobile device cansimilarly transmit the at least one picture and/or video to the capturedimage processing system 4004, and the captured image processing system4004 can automatically an alert an emergency response entity and/or sendthe diagnosis data, emergency care instructions, and/or captured imagesto a first response medical processional already en route to respond tothe emergency, to aid the first response medical processional to providespecialized care once they arrive. The bystander can utilize the sameand/or different mobile device to call 911. In some embodiments, thepictures and/or diagnosis data generated by the captured imageprocessing system 4004 can be automatically transmitted directly to anemergency response service in response to the bystander utilizing themobile device to call 911. In some embodiments, the mobile device canautomatically prompt the bystander to use the camera to take one or morepictures for automatic transmission to the captured image processingsystem 4004 in response to detecting the bystander utilized the mobiledevice to call 911.

As another example, one or more home surveillance cameras, camerasmounted on vehicles, public infrastructure surveillance cameras, and/orcameras capturing a stream of image can automatically determine if imagedata of an emergency medical condition has been captured, for example,by utilizing the captured image processing system 4004 on a stream ofimage data captured by one or more cameras. For example, a person havinga seizure, a person who is bleeding and/or has been severely wounded, afight or other dangerous activity that did or could lead to injury, aperson who has undergone an major fall or major collision, a person whois choking, or another person undergoing a medical emergency canautomatically be detected over multiple images in the video stream. Insome embodiments, single frames are processed individually, and a majorwound, injury, or emergency condition can be detected in the singleframe. When a person with an emergency medical condition isautomatically detected in a single image and/or in video, an emergencyresponse entity can automatically be alerted to the location of theemergency. The diagnosis data, captured image, and/or emergency careinstructions generated by the captured image processing system 4004 canbe transmitted to a client device associated with a first responsemedical professional routed to the location of the emergency, allowingthe first response medical professional to be prepared to givespecialized care.

While FIG. 15A depicts a single client device, any number of differentclient devices can send any number of captured images to the capturedimage processing system 4004, and the same or different one or moreclient devices can receive the diagnosis data generated by the capturedimage processing system 4004 for the captured image. As discussed inconjunction with the ECG interpretation system 4002, the diagnosis datacan be sent in accordance with an email address or messaging handleassociated with a user account utilized to transmit the captured image.The captured image processing system 4004 can store, access and/ormanage a database of user accounts which can include the same ordifferent information as the database of user accounts stored by and/oraccessed by the ECG interpretation system 4002. The database of useraccounts can similarly store current and/or historical medicalinformation for the user, user preference data for use of user interface4077, captured images and corresponding diagnosis data generated for theuser's use of the system, and/or other information determined viainteraction with user interface 4077. While FIG. 15A depictscommunication between the client device 120 and captured imageprocessing system 4004 via network 150, in other embodiments, the clientdevice can store application data for some or all of one or moreprocessing functions 4008 and/or the image classifier function 4007. Forexample, the client device 120 of FIG. 15A can locally execute theapplication data to execute ECG processing function 4005 locally oncaptured images taken by or uploaded to the client device and togenerate diagnosis data automatically in response.

FIG. 15B illustrates an example embodiment of a user interface 4077displayed via the client device 120 in conjunction with capturing and/orpreparing the captured image and/or other information for transmissionto the captured image system. Some or all of the features presentedand/or layout presented can be the same and/or different in variousembodiments of user interface 4077. The user can upload to the clientdevice and/or capture a picture via a camera of the client device togenerate the captured image for transmission to the captured imageprocessing system 4004. The captured image can be displayed to the useras depicted in FIG. 15B, and the user can enter additional informationbefore transmission, to be sent with the transmission of the photo. Theuser can elect to utilize the camera of the client device itself to takeand/or retake photos as needed, as discussed in conjunction with userinterface 4075 of FIG. 13B. In the example depicted in FIG. 15B, thecaptured image includes a picture of the user's hand with an abnormalcondition 4091.

The user can be prompted to and/or can elect to indicate the abnormalityin the captured image, for example, by interacting with the capturedimage via user interface 4077 to generate location data for the abnormalconditions by circling and/or drawing one or more segmented polygonsaround one or more portions of the abnormal condition and/or byotherwise selecting one or more pixels that correspond to the portionsof the image that correspond to the abnormal condition. For example, thelocations of two separate portions of an abnormal skin condition 4091 aspresented in FIG. 15B can each be selected by the user. The locationdata for the abnormal condition can be transmitted in conjunction withthe captured image, and can be utilized by the captured image processingsystem 4004 to localize the abnormality in the captured image. In someembodiments, this information is not entered by the user, and/or thecaptured image processing system 4004 automatically localizes theabnormality.

The user can be prompted to and/or can elect to select an anatomicalregion corresponding to the image. For example, the user can indicatethat the captured image corresponds to the user's left arm and/or canindicate that the abnormality is included on the skin of the user's lefthand. This can be utilized by the captured image processing system 4004to determine the location of the abnormality, which can aid ingenerating diagnosis data if the condition is always localized in acertain anatomical region and/or can aid in performing the imageclassifier function 4007, for example, where the captured image of FIG.15B is processed by performing a processing function 4008 trained todiagnose skin conditions in response to the user indicating the capturedimage corresponds to skin and/or where the captured image of FIG. 15B isprocessed by performing a processing function 4008 trained to processimages of hands and/or arms in response to the user indicating theabnormality is included on the left hand and/or in response to the userindicating the entire arm is included in the captured image. In someembodiments, for example, if the captured image is very zoomed in onskin and it is visually unclear which part of the body the picture istaken of, this information can be very helpful in localizing theanatomical region of the abnormality. In other embodiments, the userdoes not select an anatomical region, and/or the captured imageprocessing system 4004 automatically detects a hand in the capturedimage to determine the captured image is taken of the hand, and/or thecaptured image processing system 4004 automatically detects skin in thecaptured image to determine the captured image is taken of the skin.

The captured image can include a scale marker 4092, which as depicted inFIG. 15B, corresponds to a quarter. The scale marker 4092 can correspondto any marker of a known size and/or shape, and can have known markingsand/or other known visual features that make it easily detectable byprocessing functions of the captured image processing system 4004. Theuser interface can include instructions for selection of and/orplacement of the scale marker 4092 in the image, for example, displayinga prompt that states “Include a quarter in the image on top of the skinin the vicinity of the skin condition.” In some embodiments, the scalemarker is placed on the skin and/or otherwise on the body, and/or thescale marker can be placed next to the body within the image. In someembodiments, the scale marker 4092 is a commonplace item with knownsize, such as a quarter. In some embodiments, the scale marker 4092 is aspecific marker and/or two dimensional printed logo or other imagedelivered to the user and/or downloaded to the same or different clientdevice and printed by the user on paper in a known size. In someembodiments, the user can select from a set of scale marker options, andcan select which scale marker is included by electing to enter scaleinformation via the user interface 4077. In some embodiments, the usercan utilize any item or printed image as the scale marker, and can, forexample take other captured images of the scale marker and/or providemeasurements for the scale marker. In some embodiments, the userinterface 4077 presents size, color, and/or reflectivity requirementsfor the scale marker, and may prompt the user to utilize multiple scalemarkers that meet different sets of size, color, and/or reflectivityrequirements. In some embodiments, the user can interact with thecaptured image to select the location of the scale marker 4092 in thecaptured image, where this indicated location of the scale marker isutilized by the image classifier function 4007 and/or the processingfunction 4008 in detecting the scale marker in the image.

Performance of the image classifier function 4007, the processingfunction 4008, or another function performed on the captured image thatutilizes a computer vision model trained to detect the scale marker4092, can be utilized to detect in the captured image and/or utilize thescale marker 4092. The captured image processing system 4004 can utilizeknown dimensions and/or known size of the scale marker 4092 to determineestablished scale information for the captured image, for example, basedon dimensions and/or size of the scale marker 4092 in relation to thecaptured image. The established scale information can be utilized ingenerating diagnosis data, in performing one or more lesion measurementfunctions, and/or to determine size and/or portions of the body includedin the image. Alternatively or in addition, the image classifierfunction 4007 and/or the processing function 4008 can determine pixelcolor data of the scale marker detected in the captured image, and cancompare the pixel color data to known color information and/or knownreflectivity information of the scale marker, for example, to establishlighting conditions in the captured image and/or to otherwise establishbaseline lighting and/or color data for the image. In particular, ifcolor is an important feature in diagnosing the abnormal condition, theestablished lighting and/or color information can be utilized todetermine how the colors and/or brightness in the captured image couldbe distorting true color and/or brightness of the abnormal condition.

In various embodiments, the pre-processing of the image data includesadjusting color and/or brightness in the entire image by a factornecessary to adjust the color and/or brightness of the scale marker inthe captured image to a preset baseline color and/or brightness.Alternatively or in addition, the pre-processing of the image dataincludes adjusting scale of the entire image by a factor necessary toadjust the size of the scale marker in the captured image to a knownbaseline size.

In some embodiments, the client device can locally determine whether thescale marker is detected in the captured image before transmission,and/or can determine if the resolution, size, color, and/or brightnessof the scale marker as presented in the image indicates that lightingconditions, resolution, sizing, and/or other conditions of the photowill be sufficient for processing. In some embodiments, the user can beprompted to retake the photo if the size of the scale marker as detectedin in the captured image compares unfavorably to a threshold, if theresolution and/or sharpness of the scale marker as detected in in thecaptured image compares unfavorably to a threshold, and/or if thebrightness and/or color of the of the scale marker as detected in in thecaptured image compares unfavorably to a threshold. In some embodiments,the user is prompted with instructions to correct the problem, forexample, where a prompt such as “please retake the photo in a brighterroom” is presented in response to the brightness and/or color of the ofthe scale marker as detected in in the captured image comparingunfavorably to the threshold.

Instead of or in addition to utilizing a scale marker 4092, the user canmanually enter information via user interface 4077 that can be utilizedto determine the established scale data and/or the established colorand/or brightness data. For example, the user can be prompted to measureone or more dimensions of or more abnormalities or other features in thecaptured by utilizing their own measuring device. The user can indicateendpoints of the corresponding measured dimensions in the captured imageby interacting with the captured image via the user interface 4077, andthe captured image processing system 4004 can utilize this informationto determine the established scale data. The user can be presented witha plurality of color options and can select one or more color that mostclosely matches one or more colors of the abnormalities when viewed withthe naked eye. The user can indicate locations of each one or more colorof the abnormality in the captured image data by interacting with thecaptured image via the user interface 4077, and the captured imageprocessing system 4004 can utilize this information to determine theestablished color data and/or brightness data.

The user can enter additional observations and/or symptoms to be sent tothe captured image processing system 4004. For example, the user canenter natural language text describing observations and the capturedimage processing system 4004 can perform a natural language processingfunction can utilize a natural language model, for example, by utilizingthe medical scan natural language analysis system 114, to extract keyterms from and/or interpret some or all of the natural language text.Alternatively or in addition, the user interface can present a series ofquestions to the user, and the answers of the series of questions can betransmitted to the captured image processing system 4004. In someembodiments, a decision tree of prompts can be stored locally by theclient device and/or can be utilized by the captured image processingsystem 4004 to determine the next question based on one or more answersto one or more previous questions, for example, where each answer istransmitted to the captured image processing system 4004 and the nextquestion is received from the captured image processing system 4004based on the previous answer. For example, questions such as “does theskin condition burn?”, “does the skin condition itch?”, “does the skincondition feel painful when pressure is applied?”, “how long has theskin condition been visible?”, “how much larger has the skin conditiongrown in the last 24 hours?”, and/or other questions presented to theuser to provide additional information utilized to generate diagnosisdata.

Alternatively or in addition, the user can be presented with drawingsand/or captured images of abnormal conditions that the user's conditionmay correspond to, based on answers to questions so far, and the usercan be prompted to select one or more images that look most similar totheir condition. The known diagnosis for these images can be utilized togenerate diagnosis data for the user. The answers to these questions canalso be utilized to determine established scale information and/orestablished color information and/or brightness information for thecaptured image received from the user.

In some embodiments, some or all of the questions are determined by thecaptured image processing system 4004 after the captured image has beenreceived and after diagnosis data has been generated, for example, wherethe questions are determined based on findings in the diagnosis dataand/or missing information from the diagnosis data. In some embodiments,the user is prompted to take additional captured images, for example,with correction instructions based on problems with the prior capturedimage and/or with instructions to take pictures of additional areas ofthe body based on the diagnosis. The user can be prompted to wait afixed amount of time, such as 24 hours, to retake one or more pictures,for example, where a measurement change function can be applied to theabnormal condition detected in the images over time to determine if theabnormal condition is growing and/or shrinking with time, if the colorof the abnormal condition is changing with time, and/or if and/orbecoming more or less severe. This can be utilized by the captured imageprocessing system 4004 to generate additional diagnosis data for thepatient after initial diagnosis data has already been generated andtransmitted.

FIG. 15C illustrates an example of embodiment of user interface 4077presented to the user in response to receiving the diagnosis data fromthe captured image processing system 4004. The user interface 4077 ofFIG. 15C can include some or all of the features of the user interface4075 discussed in conjunction with FIG. 13D and/or 13E. Some or all ofthe features of user interface 4077 presented in FIG. 15C can bepresented to medical professionals, for example, if the captured imageand/or diagnosis data is sent to a medical professional providing careto the user, a medical professional selected to provide a secondopinion, and/or a medical professional providing additional diagnosisdata utilized by the captured image processing system 4004 to generatefinal diagnosis data.

The user interface 4077 can display diagnosis data for exampleindicating a diagnosis of Impetigo as shown in FIG. 15C. The display ofdiagnosis data can also include additional information such as treatmentinstructions, what symptoms to expect, a length of time expected for theabnormality to clear if treatment is applied as instructed, and/or otherinformation. The user can be instructed to receive medical attention inperson based on an urgency of the diagnosis data, and one or more localphysicians and/or hospitals can be automatically determined by theclient device and/or the captured image processing system 4004 based ona determined location of the client device, based on an address of theuser in the user's account, and/or based on health insurance informationprovided by the user. The physicians and/or hospitals can be filtered byspecialty based on the diagnosis data, for example, where onlydermatologists and/or dermatology facilities are presented to the userin response to the diagnosis data indicating a skin condition that needsmedical attention.

The user interface 4077 can present the captured image in conjunctionwith the diagnosis in a processed and/or unprocessed format. Forexample, the captured image can be automatically cropped and/orbrightness, contrast, and/or other filters may have been automaticallyapplied by the captured image processing system 4004 to better enhancethe abnormality in the image. The user interface 4077 can displayannotation data overlaid upon the diagnosis data, indicating locationsof automatically detected features in the captured image, indicatingautomatically generated measurements of one or more lesions or otherabnormal conditions in the captured image, and/or indicatingautomatically generated characterization data of the one or moreabnormal conditions in the captured image. In embodiments where multiplecaptured images and/or a video was utilized, a subset of captured imageand/or a subset of frames of the video can be displayed, for example asstill frames. In some embodiments a change in the abnormality over timecan be visually displayed based on displaying corresponding capturedimages side by side and/or based on displaying multiple measurementstaken over time overlaid upon a single image, for example, as discussedin conjunction with the lesion tracking system of FIGS. 16A-16D.

The user interface 4077 can allow the user to elect to transmit thediagnosis data to a designated physician or other user indicated in auser account associated with the user and/or entered by the user viauser interface 4077 or other user indicated by the user. The userinterface 4077 can allow the user to elect to transmit a prescriptionautomatically generated in the diagnosis data and/or received from amedical professional to a pharmacy indicated by the user via userinterface 4077 and/or in the user account. In some embodiments, thediagnosis data can indicate that over the counter medication and/orother commercially available tools and/or devices can be utilized totreat the condition at home. For example, the diagnosis data canindicate that tweezers can be utilized to remove a splinter detected inthe captured image and/or can indicate that a salicylic acid toner canbe utilized to treat acne detected in the captured image. Links toproducts corresponding to recommended medication, tools, and/or devicescan be provided to the user and/or the user can be prompted to purchaseone or more recommended items for home treatment via user interface4077.

The user interface 4077 can allow the user to elect to generate aformatted report such as PDF report that includes the original and/orprocessed captured image with or without annotations, and/or thatfurther includes text data generated in the diagnosis data. The userinterface 4077 can allow the user to request a second opinion, and thediagnosis data and/or captured image can be transmitted to a medicalprofessional selected automatically and/or selected by the user inresponse for display to the selected medical professional via userinterface features discussed in conjunction with user interface 4076.The user interface 4077 can allow the user to view additionalinformation about the diagnosis, and/or can present links and/orarticles to the user for display in via user interface 4077 and/or a webbrowser. While these features are presented as options to the user inFIG. 15C, some or all of these features can be performed automaticallyby the client device and/or the captured image processing system 4004 inresponse to the diagnosis data being generated for the captured image.

FIG. 16A illustrates an embodiment of a lesion tracking system 3002. Thelesion tracking system 3002 can receive multiple scans or otherlongitudinal of the same patient to track changes in one or more lesionsdetected in the multiple scans over time. In particular, the lesionsize, shape, diameter, and/or volume, and/or other characteristics ofthe lesion such as other abnormality classification data 445 can bedetermined for each scan, and the changes in these features over timecan be measured and tracked. For example, lesions can be determined toshrink, grow, or disappear over subsequent medical scans, and/or newlesions can be detected to appear over subsequent medical scans.Performing such calculations automatically by utilizing the lesiontracking system 3002 can generate more precise measurements than thosegenerated by a radiologist's visual inspection of one or more medicalscans. These automated measurements can thus be used to more accuratelydetermine or predict if a patient's condition is bettering or worsening,to more accurately determine or predict if a patient is responding wellor poorly to treatment, and/or to otherwise aid in diagnosing apatient's condition.

As shown in FIG. 16A, lesion tracking system 3002 can communicatebidirectionally, via network 150, with the medical scan database 342and/or other databases of the database storage system 140, with one ormore client devices 120, and/or, while not shown in FIG. 16A, one ormore subsystems 101 of FIG. 1. In some embodiments, the lesion trackingsystem 3002 is an additional subsystem 101 of the medical scanprocessing system 100, implemented by utilizing the subsystem memorydevice 245, subsystem processing device 235, and/or subsystem networkinterface 265 of FIG. 2A. In some embodiments, the lesion trackingsystem 3002 is implemented by utilizing, or otherwise communicates with,the central server 2640. For example, some or all of the databases ofthe database storage system 140 are populated with de-identified datagenerated by the medical picture archive integration system 2600. Insome embodiments, the lesion tracking system 3002 can receivede-identified medical scans, annotation data, and/or reports directlyfrom the medical picture archive integration system 2600. For example,the lesion tracking system 3002 can request de-identified medical scans,annotation data, and/or reports that match requested criteria, forexample, corresponding to training set criteria. In some embodiments,some or all of the lesion tracking system 3002 is implemented byutilizing other subsystems 101 and/or is operable to perform functionsor other operations described in conjunction with one or more othersubsystems 101.

The lesion tracking system 3002 can be operable to receive, viasubsystem network interface 265 or another receiver, a first medicalscan that is associated with a first unique patient ID and a first scandate. The lesion tracking system 3002 can also receive a second medicalscan that is associated with the first unique patient ID and a secondscan date that is different from the first scan date. The first medicalscan can include a first plurality of image slices, and the secondmedical scan can include a second plurality of image slices. As shown inFIG. 16A, the first medical scan and second medical scan can be receivedas medical scan entries 3005 and 3006, respectively. The medical scanentries can be received from the medical scan database 342, and eachentry can include some or all fields of medical scan entries 352 asdescribed in conjunction with FIG. 4A. For example, the unique patientID can be indicated in the patient identifier data 431 and/or the scandate can be indicated in the scan date data 426. In some embodiments,more than two medical scans of the patient can be received forprocessing. In some embodiments, medical scan entry 3006 can be receivedas longitudinal data 433 of medical scan entry 3005 and/or an identifierof medical scan entry 3006 can be determined from longitudinal data 433of medical scan entry 3005, which can be utilized by the lesion trackingsystem to fetch medical scan entry 3006 from the medical scan database342. Medical scan entries 3005 and 3006 can correspond to the same ordifferent scan categories, and can, for example, correspond to the sameor different modality.

A lesion detection function 3020 can be performed to detect at least onelesion in medical scan entries 3005 and 3006. In some embodiments, thelesion detection function 3020 is performed on image data 410 on medicalscan entries 3005 and 3006 to determine an anatomical location of thelesion, to determine a subset of image slices that contains the lesionfor each medical scan, to determine abnormality location data 443corresponding to the lesion, and/or to otherwise determine the locationof the lesion in the image data. For example, as depicted in FIG. 16A,image slice subset 3030 can correspond to the subset of slices thatinclude the detected lesion in image data 410 of medical scan entry3005, and image slice subset 3031 can correspond to the subset of slicesthat include the detected lesion in image data 410 of medical scan entry3006.

In some embodiments, the lesion detection function 3020 is implementedby utilizing a medical scan analysis function, for example, trained bythe medical scan image analysis system 112. In such embodiments, thelesion detection function can correspond to the inference step 1354and/or the detection step 1372 described in conjunction with FIG. 7B, todetermine abnormality region 1373. In some embodiments, the lesion isdetected in an image slice of the image data 410, A density value,density range and/or other pixel value of pixels determined tocorrespond to the lesion in the image slice is determined. This densityvalue, density range and/or other pixel value is compared to the valuecorresponding pixels in neighboring image slices, or pixels withinproximity of coordinate values determined to contain the lesion in theimage slice. For example, the neighboring image slices can include oneor more image slices before or after the image slice in the sequentialslice ordering of the image data. If the pixel values compare favorably,this can be utilized to determine that the lesion is included in theseneighboring slices and/or to determine which pixels of the neighboringimage slices include the lesion. This process can continue forsubsequent neighboring image slices to determine the remainder of theimage slice subset 3030, continuing until no more neighboring imageslices are determined to include the lesion. Thus, the image slicesubset 3030 can correspond to a consecutive subset of image slices withrespect to the sequential ordering of the image slices of the image data410.

In some embodiments, the lesion detection function 3020 is firstperformed on medical scan entry 3005, and the anatomical location and/orsubset of image slices is utilized to detect the lesion in medical scanentry 3006, for example, to ensure the same lesion is detected in bothmedical scan entries and/or to expedite processing of medical scan entry3006. For example, performing the lesion detection function on medicalscan entry 3006 can include searching only a subset of image slices ofthe medical scan entry 3006 corresponding to and/or neighboring theimage slice subset 3030; searching an anatomical region determined inprocessing medical scan entry 3005 for the lesion; and/or searching onlya subset of pixels of some or all image slices corresponding to and/orin proximity to the anatomical region, and/or pixels of the image slicesubset 3030 determined to include the lesion. In some embodiments, thelesion detection function includes performing an abnormality similarityfunction or other medical scan similarity analysis function trained byand/or performed by the medical scan comparison system 116, where asimilarity score for lesions detected in medical scan entry 3005 and3006 is compared to a threshold, and is utilized to determine that thedetected in medical scan entry 3006 is the same lesion as that detectedin medical scan entry 3005 when the similarity score compares favorablyto a threshold.

Once the lesion is detected, the image slice subset 3030, anatomicalregion data, pixel coordinates corresponding to the detected lesion,and/or other abnormality location data 443 corresponding to the lesioncan be utilized as input to one or more lesion measurement functions3045. In some embodiments, the lesion detection function 3020 is notperformed by the lesion tracking system 3002. Instead, abnormalitylocation data 443 that indicates the subset of the image slice subset3030 and/or 3031, anatomical region data, pixel coordinatescorresponding to the detected lesion, and/or other location data can bereceived from the medical scan database 342 and/or another subsystem 101for use as input to the lesion measurement function 3045.

The one or more lesion measurement functions 3045 can include a lesiondiameter measurement function, as shown in FIG. 16B, to determinediameter measurement 3022 for a lesion 3010 detected in the image data410 of medical scan entry 3005 and/or to determine a diametermeasurement 3024 for the lesion 3010 detected in image data 410 ofmedical scan entry 3006.

For a lesion 3010 detected in the image data of medical scan entry 3005,the lesion diameter measurement function can include performing a lesiondiameter calculation on each of the image slice subset 3030 to generatea set of diameter measurements. Generating the lesion diametermeasurement for the lesion of medical scan entry 3005 can includeselecting a maximum of the set of diameter measurements. The lesiondiameter measurement can correspond to a segment connecting a firstpoint and a second point of a perimeter of the lesion in one of theimage slice subset 3030. In some embodiments, the segment is oblique toan x-axis of the one of the image slice subset. In some embodiments,performing the lesion diameter measurement function can includedetermining a set of pixels of some or all of the subset of image slicesthat correspond to the perimeter of the first lesion in the one of thefirst subset of image slices. A set of segment lengths corresponding toa distance between each of a plurality of pairs of pixels can becalculated, for example, where the plurality of pairs of pixels includesevery combination of selecting two of the set of pixels. The lesiondiameter measurement can be determined by selecting a maximum of the setof segment lengths.

The diameter measurement 3024 corresponding to the diameter of thelesion 3010 in the image data of medical scan entry 3006 can becalculated in the same or different fashion. The diameter measurement3024 can correspond to a segment on the same image slice index ordifferent image slice index of the image slice that includes thediameter measurement 3022 for medical scan entry 3005. For example, theimage slice containing the diameter of the lesion may change dependingon how the lesion changed shape over time. Similarly, the axis alongwhich the diameter falls relative to a coordinate system of the imageslices can be different for diameter measurements 3022 and 3024, asshown in FIG. 16B.

In some embodiments, the diameter measurement can be measured acrossmultiple slices, for example, based upon the three-dimensional structureof the lesion. For example, segment lengths for a plurality of pairs ofpixels corresponding to the three-dimensional surface of the lesionacross some or all of the image slice subset 3030 can be utilized tocompute the diameter measurement 3022. In particular, a slice thicknesscan be determined, for example, based on metadata of the medical scanentry 3005 and/or based on the modality of the medical scan entry 3005,and can be used in computing the segment lengths for each of theplurality of pairs. The maximum segment length can be utilized as thediameter measurement 3022.

In some embodiments, the one or more lesion measurement functions 3045can include a lesion area measurement function. For example, based onpixels in each of the subset of image slices determined to be includedin the lesion, an area can be computed. In particular, a fixed pixelarea corresponding to the true area represented by each individual pixelcan be determined, for example, in the medical scan entry metadataand/or based on the modality of the medical scan. This pixel area can bemultiplied by the number of pixels determined to be included in thelesion to calculate a lesion area for each image slice in the imageslice subset.

Furthermore, this calculated set of areas can be utilized to calculate avolume approximation of the lesion by performing a lesion measurementfunctions 3045 corresponding to a lesion volume measurement function.Performing the lesion volume measurement function can include performinga Riemann sum calculation on the set of lesion area measurements, wherea uniform partition width of the Riemann sum is determined based on thedetermined slice thickness of the image slices in the image data. Forexample, every pair of consecutive image slices of the image slicesubset 3030 can correspond to one of a plurality of trapezoidal areas.Performing the performing the lesion volume calculation can includeperforming a summation of the plurality of trapezoidal areas. Each ofthe plurality of trapezoidal areas can be calculated by multiplying theslice thickness by half of the sum of a first base and a second base,where a value of the first base is equal to a first one of the set oflesion area measurements corresponding to a first one of a correspondingpair of consecutive image slices, and where a value of the second baseis equal to a second one of the of the set of lesion area measurementscorresponding to a second one of the corresponding pair of consecutiveimage slices.

FIG. 16C illustrates an example of performing the lesion volumemeasurement function. Image slice subset 3030 is determined from theimage data 410 based on the detection of lesion 3010, and includes sliceindexes 0-10. The lesion area of lesion 3010 can be calculated for eachimage slice, as illustrated in the discrete plot 3032 of slice index vslesion area. Plot 3032 can be utilized to determine volume as the areaunder the curve of plot 3034 to perform a trapezoidal Riemann sumapproximation of lesion volume, where the x-axis measurescross-sectional distance, or width, from slice 0. This can be determinedby multiplying the slice index of the x-axis of plot 3032 by the slicethickness to determine the x-value of each of the coordinates plotted inplot 3032. A continuous curve of lesion area can be approximated byconnecting discrete points of plot 3032 to create the curve of plot3034. While linear segments are shown to connect the discrete points inFIG. 16C, any curve fitting function can be utilized to generate thearea curve. In this example, calculating the area under the curve toapproximate volume can correspond to a trapezoidal Riemann sumapproximation, but other Riemann sum approximations, other integralapproximation functions, and/or other volume approximation techniquescan be utilized to approximate volume based on the discrete areas ofplot 3032.

One or more of the lesion measurement functions 3045 can be medical scananalysis functions, for example, trained by and/or performed by themedical scan image analysis system 112 and/or trained and/or performedin the same fashion as other medical scan analysis functions describedherein. In some embodiments, the lesion measurement function isimplemented by utilizing the abnormality classification step 1374 togenerate classification data 1375 that includes the lesion measurementdata 3040 and/or 3041.

The lesion measurements can be compared by performing a lesionmeasurement change function 3050 on the lesion measurement data 3040 and3041. The lesion measurement change function 3050 can include computingdifference of corresponding measurement values, such as a difference indiameter and/or a difference in volume of the lesion. The lesionmeasurement function can also calculate a Euclidean distance of vectorsthat include a set of measurements in lesion measurement data 3040 and3041. The lesion measurement change function 3050 can be a medical scananalysis function, such as a medical scan comparison function, trainedby and/or performed by the medical scan image analysis system 112,trained by and/or performed by the medical scan comparison system 116,and/or trained and/or performed in the same fashion as other medicalscan analysis functions described herein.

In some embodiments, the lesion measurement function 3045 is notperformed by the lesion tracking system 3002. Instead, abnormalityclassification data 445 corresponding to one or more measurementcategories 444 can include lesion measurement data 3040 and/or 3041, andcan be received from the medical scan database 342 and/or anothersubsystem 101 for use as input to the lesion measurement change function3050.

The lesion measurement change data 3055 can be transmitted via subsystemnetwork interface 265 and/or via another transmitter, for transmissionto one or more client devices 120 for display via a display device. Forexample, the lesion measurement change data can be displayed as textand/or can be displayed visually in conjunction with the image data 410of medical scan entries 3005 and/or 3006 by utilizing the medical scanassisted review system 102. For example, the measurement data can bedisplayed as state change data of abnormalities detected in longitudinaldata as described in conjunction with the of the medical scan assistedreview system 102. Alternatively or in addition, the lesion measurementchange data 3055 can be sent to one or more other subsystems forprocessing, for example, to be utilized as training data by one or moremedical scan analysis functions trained by medical scan image analysissystem 112. Alternatively or in addition, the lesion measurement changedata 3055 can be sent to the medical scan database for storage, forexample, as part of the longitudinal data 433 for medical scan entry3005 and/or 3006. Alternatively or in addition, the lesion measurementdata 3040 and/or 3041 can be sent to the medical scan database forstorage, for example, as part of abnormality classification data 445 formedical scan entry 3005 and/or 3006, respectively, corresponding toabnormality classifier categories 444 corresponding to a diametercategory, an area category, a volume category, or other measurementcategory.

In some embodiments, a set of three or more medical scans of the samepatient are received, and the lesion measurement change data iscalculated for consecutive ones of the set of three or more medicalscans with respect to scan data. In some embodiments, lesion measurementchange data is also calculated for some or all of every possible pair ofthe medical scans in the set of three or more medical scans.

FIG. 16D illustrates an example of an interface 3080, which can bedisplayed on a display device of client device 120. The interface canpresent a selected image slice of each image slice subset 3030 and 3031.A region detected to include the lesion can be overlaid on the imageslice as annotation data, and/or other annotation data can be displayedto indicate the lesion. In some embodiments, the diameter measurementdata can be displayed visually for medical scan entries 3005 and/or3006. For example, the image slice of image slice subset 3030 and/or3031 determined to include the largest diameter can be automaticallypresented, and a segment connecting the corresponding first pixel andsecond pixel determined to correspond to endpoints of the diameter canbe automatically overlaid on the displayed image slice. In someembodiments, a solid or semi-transparent outline and/or shading of thepixels determined to include the lesion in an image slice of medicalscan entry 3005 can be overlaid upon the corresponding pixel coordinatesin the display of the corresponding image slice of medical scan entry3006 by the interface, for example, to visually depict how much thelesion has shrunk, grown, or otherwise changed shape and/or position. Insome embodiments, some or all of the lesion measurement data and/orlesion measurement change data is displayed as text in conjunction withthe image data. In some embodiments, a three-dimensional rendering ofthe lesion, generated based on the lesion volume measurement data, canbe displayed in accordance with a three-dimensional visualizationinterface.

In some embodiments, other classification data can be generated based ona diameter measurement, area measurement, and/or volume measurement ofthe lesion measurement data. For example, the lesion diameter data canbe utilized to determine RECIST eligibility data and/or can be utilizedto determine whether or not the lesion corresponds to a target lesion ornon-target lesion. The lesion change measurement data can be utilized todetermine RECIST evaluation data based on the change in the lesion in amore recent scan when compared to a prior scan. In particular, thelesion change measurement data can be utilized to indicate if the lesionis “Complete Response”, “Partial Response”, “Stable Disease”, or“Progressive Disease”. In cases where three or more scans are evaluatedfor a patient, the RECIST evaluation data can reflect changes over time.In some embodiments, a plurality of lesions are detected, measured andtracked in the medical scan entries 3005 and 3006. RECIST eligibilitydata and/or RECIST evaluation data can be generated for each theplurality of lesions, and/or RECIST evaluation data and/or diagnosisdata can be generated based on assessing the plurality of lesions as awhole.

RECIST eligibility data and/or evaluation data can be transmitted to theclient device for display via the display device, can be transmitted tothe medical scan database for storage in a corresponding medical scanentry as abnormality annotation data 442 and/or as longitudinal data433, and/or can be transmitted to other subsystems 101, for example, aspart of a training set to train a medical scan analysis function. Otherstandardized medical assessment scores characterizing the lesion, suchas a Lung-RADS assessment score, can be generated automatically based onthe measurement data.

The medical scan entries 3005 and 3006 can be received at the same timeor different times for processing. For example, as medical scan entries3005 and 3006 correspond to different scan dates, they can be sent tothe medical scan lesion tracking system for processing as scans aretaken for the patient. In some embodiments, only medical scan entry 3005is received, and lesion measurement data is calculated for medical scanentry 3005. This can be sent to the client device 120 and/or can be sentto the medical scan database 342 for storage as abnormality annotationdata 442 or other data of the medical scan entry 3005. Later, medicalscan entry 3006 can be received, and lesion location data and/or lesionmeasurement data 3040 corresponding to the lesion in medical scan entry3005 can be fetched from the database in response to generate the lesionmeasurement change data 3055. The lesion location and/or measurementdata of the lesion in medical scan entry 3005 can also be utilized toaid in detecting the lesion in medical scan entry 3006, to aid ingenerating lesion measurement data for the lesion in medical scan entry3006.

In some embodiments, the data generated by the lesion tracking system3002 can be utilized to train a longitudinal lesion model. Thelongitudinal lesion model can be generated by the lesion tracking model,and/or output of the lesion tracking model can be sent to anothersubsystem, such as the medical scan image analysis system. For example,a training step 1352 can be performed on a plurality of sets oflongitudinal data, where each set of longitudinal data corresponds to apatient and includes the lesion measurement data, the lesion measurementchange data, the classification data such as RECIST eligibility data,RECIST evaluation data, and/or Lung-RADS assessment data determined fora corresponding plurality of medical scans entries of the patient. Eachof the plurality of sets of longitudinal data can include other fieldsof the corresponding plurality of medical scan entries of the patient,such as the image data, diagnosis data, patient history, and/or otherrelevant fields of one or more medical scan entries of the correspondingpatient.

The longitudinal lesion model can be utilized to perform an inferencefunction on subsequent medical scans, such as a single medical scanentry of a new patient or a set of medical scan entries of a newpatient. The inference function can be performed by the lesion trackingsystem 3002, by the medical scan image analysis system 112, and/or byanother subsystem 101. The inference function corresponding to thelongitudinal lesion model can be a medical scan analysis function, andcan be trained and/or performed as discussed herein with regards tomedical scan analysis function.

By performing the inference function on one or more medical scans of apatient, lesion change prediction data can be generated for at least onelesion detected in the one or more medical scans. For example, thelesion change prediction data can include a lesion growth factor or alesion shrinkage factor. Alternatively or in addition, the inferencefunction can generate other inference data, such as other assessmentand/or prediction data. This can include inference data that assesseslesion growth and/or shrinkage in the set of medical scans, thatassesses and/or predicts changes in the severity of the patient'scondition, that diagnoses the new patient, that includes determinedtreatment steps for the new patient, that determines whether the newpatient is responding favorably or unfavorably to treatment, and/or thatotherwise assesses and/or predicts the new patient's current conditionand/or future condition. Some or all of the inference data generated byperforming the inference function can be determined based on assessingthe size and/or characteristics of detected lesions, and/or based onpredicting the change in size or change in characteristics of detectedlesions.

In some embodiments, performing the inference function includesperforming the lesion measurement function on the one or more medicalscans of the new patient and/or includes performing the lesionmeasurement change function on the one or more medical scans of the newpatient. The lesion measurement data and/or lesion measurement changedata generated for the new patient can be input to the inferencefunction in addition to or instead of the one or more medical scansentries themselves.

The lesion change prediction data or other inference data can betransmitted to a client device for display on a display device via aninterface, for example, in conjunction with the one or more medicalscans of the new patient. Presenting the lesion change prediction datacan include overlaying a predicted diameter, area, and/or volume changeof the lesion, for example, by displaying a solid or semi-transparentoutline and/or shading of the pixels in accordance with a predictedfuture size, a predicted future shape, and/or predicted future locationof the lesion in at least one image slice of the one or more new medicalscan entries, to visually depict how much the lesion is predicted toshrink, grow, or otherwise change shape and/or position. In someembodiments, a predicted future three-dimensional rendering of thelesion can be displayed in accordance with a three-dimensionalvisualization interface.

In some embodiments, the inference function can generate a set of lesionchange prediction data corresponding to a set of different projectedtime spans. For example, lesion change prediction data can be generatedfor one year, two years, and three years in the future, and theprediction data for each projected time span can be sent to the clientdevice for display. In some embodiments, the interface can prompt theuser to select one of the set of different projected time spans, and theprediction data for the selected one of the projected time spans will bedisplayed accordingly. To enable this capability, the longitudinallesion model can be trained on sets of longitudinal data with medicalscans of varying time spans, and the relative time between dates ofmedical scans and/or dates of other data in a set of longitudinal datacan be utilized in performing the training step.

In some embodiments, before execution of the inference function on theone or more medical scans of the new patient, a user interacting withthe interface displayed by the display device can select a projectedtime span from a discrete set of options, and/or can enter any projectedtime span. The inference function can be performed by utilizing theselected projected time span received from the client device, andprediction data can reflect this selected projected time span from thecurrent date and/or from a date of the most recent scan in the one ormore medical scans for the new patient. For example, if the selectedprojected time span is 18 months, the inference data can include alesion growth factor, a lesion shrinkage factor, and/or other predictiondata projected for 18 months in the future.

In some embodiments, medical scan entry 3005 and/or medical scan entry3006 already have associated measurement data. Human assessment data,such as human measurement data corresponding to a radiologistmeasurement or other human measurement of the lesion, can be included inthe medical scan entry and/or can be received in conjunction with themedical scan entry. For example, a human diameter measurement can beincluded in the human assessment data of a medical scan corresponding toa radiologist's documentation of the diameter based on visual inspectionof the image data of the medical scan. This human assessment data cancorrespond to abnormality annotation data 442 with diagnosis author data450 corresponding to the radiologist or other human that took themeasurement. This diagnosis author data 450 can correspond to anidentifier of the radiologist or other human in a corresponding userprofile entry 354 the user database 344. The human assessment data canalso include abnormality classification data 445, such as RECISTeligibility data, RECIST evaluation data, a Lung-RADS assessment score,or other abnormality classification data 445 discussed herein.

Performing one or more of the lesion measurement functions on themedical scan entry 3005 and/or 3006 can be further utilized to measurethe accuracy of the human assessment data taken by a radiologist. Forexample, a radiologist may have measured a diameter incorrectly byfailing to measure the distance between two points of the perimeter ofthe lesion properly, by identifying a wrong segment on an image slice asbeing the maximum segment connecting perimeter points of the lesion, byidentifying a maximum segment in an image slice when a different imageslice includes a portion of the lesion with a larger maximum segment, byconsidering pixels of an image slice that are not part of the lesion ordo not correspond to the perimeter of the lesion when determining thediameter, by failing to consider a true diameter that connects twopoints along the surface of a three-dimensional representation of thelesion where the two points are on different image slices of the medicalscan, by mischaracterizing the scan and taking a measurement for alesion that is not actually a lesion, by mischaracterizing the scan andfailing to take a measurement for a lesion based on a determination thatthe lesion did not exist or based on a determination that the lesiondoes not meet criteria such as RECIST criteria, by characterizing alesion as a target lesion or non-target lesion improperly, bycharacterizing a lesion or the medical scan as “Complete Response”,“Partial Response”, “Stable Disease”, or “Progressive Disease”improperly, by determining abnormality classification data 445incorrectly, by otherwise measuring and/or characterizing the lesionimproperly, and/or by otherwise measuring and/or characterizing a changein the lesion across multiple medical scans of the patient improperly.

The accuracy of human assessment data can be determined by generatingautomated assessment data. The automated assessment data can begenerated by performing the lesion detection function, by performing theone or more lesion measurement functions, and/or by classifying thelesion, for example, by performing abnormality classification step 1374.The lesion location determined in the detection data, the lesiondiameter, area and/or volume determined in the lesion measurement data,and/or abnormality classification data 445 for one or more abnormalityclassifier categories 444 can be compared to corresponding portions ofthe human assessment data by performing a similarity function, bycomputing a difference in values, by determining whether or not thevalues match or otherwise compare favorably, and/or by computing aEuclidean distance between feature vectors of the human assessment dataand the automated assessment data.

The difference between some or all of the human assessment data and theautomated assessment data can be compared to a threshold to determine ifthe human assessment data is correct or incorrect. The differencebetween some or all of the human assessment data and the automatedassessment data can also correspond to accuracy data such as an accuracyscore, and the accuracy score can be assigned to the correspondingradiologist or other human. For example, the accuracy score can bemapped to the radiologist in the corresponding user profile entry 354 ofthe user database 344. The accuracy score can also be transmitted to aclient device for display via the display device. Accuracy scores thatcompare unfavorably to a threshold can be utilized to automatically flagradiologists or other humans that recorded an incorrect measurement orcharacterization of a lesion, and/or are consistently recordingincorrect measurements or characterizations of lesions.

As may be used herein, the terms “substantially” and “approximately”provides an industry-accepted tolerance for its corresponding termand/or relativity between items. Such an industry-accepted toleranceranges from less than one percent to fifty percent and corresponds to,but is not limited to, component values, integrated circuit processvariations, temperature variations, rise and fall times, and/or thermalnoise. Such relativity between items ranges from a difference of a fewpercent to magnitude differences. As may also be used herein, theterm(s) “configured to”, “operably coupled to”, “coupled to”, and/or“coupling” includes direct coupling between items and/or indirectcoupling between items via an intervening item (e.g., an item includes,but is not limited to, a component, an element, a circuit, and/or amodule) where, for an example of indirect coupling, the intervening itemdoes not modify the information of a signal but may adjust its currentlevel, voltage level, and/or power level. As may further be used herein,inferred coupling (i.e., where one element is coupled to another elementby inference) includes direct and indirect coupling between two items inthe same manner as “coupled to”. As may even further be used herein, theterm “configured to”, “operable to”, “coupled to”, or “operably coupledto” indicates that an item includes one or more of power connections,input(s), output(s), etc., to perform, when activated, one or more itscorresponding functions and may further include inferred coupling to oneor more other items. As may further be used herein, the term “associatedwith”, includes direct and/or indirect coupling of separate items and/orone item being embedded within another item. As may still further beused herein, the term “automatically” refers to an action causeddirectly by a processor of a computer network in response to atriggering event and particularly without human interaction.

As may be used herein, the term “compares favorably”, indicates that acomparison between two or more items, signals, etc., provides a desiredrelationship. For example, when the desired relationship is that signal1 has a greater magnitude than signal 2, a favorable comparison may beachieved when the magnitude of signal 1 is greater than that of signal 2or when the magnitude of signal 2 is less than that of signal 1. As maybe used herein, the term “compares unfavorably”, indicates that acomparison between two or more items, signals, etc., fails to providethe desired relationship.

As may also be used herein, the terms “processing module”, “processingcircuit”, “processor”, “processing device” and/or “processing unit” maybe a single processing device or a plurality of processing devices. Sucha processing device may be a microprocessor, micro-controller, digitalsignal processor, graphics processing unit, microcomputer, centralprocessing unit, field programmable gate array, programmable logicdevice, state machine, logic circuitry, analog circuitry, digitalcircuitry, and/or any device that manipulates signals (analog and/ordigital) based on hard coding of the circuitry and/or operationalinstructions. The processing module, module, processing circuit, and/orprocessing unit may be, or further include, memory and/or an integratedmemory element, which may be a single memory device, a plurality ofmemory devices, and/or embedded circuitry of another processing module,module, processing circuit, and/or processing unit. Such a memory devicemay be a read-only memory, random access memory, volatile memory,non-volatile memory, static memory, dynamic memory, flash memory, cachememory, and/or any device that stores digital information. Note that ifthe processing module, module, processing circuit, and/or processingunit includes more than one processing device, the processing devicesmay be centrally located (e.g., directly coupled together via a wiredand/or wireless bus structure) or may be distributedly located (e.g.,cloud computing via indirect coupling via a local area network and/or awide area network). Further note that if the processing module, module,processing circuit, and/or processing unit implements one or more of itsfunctions via a state machine, analog circuitry, digital circuitry,and/or logic circuitry, the memory and/or memory element storing thecorresponding operational instructions may be embedded within, orexternal to, the circuitry comprising the state machine, analogcircuitry, digital circuitry, and/or logic circuitry. Still further notethat, the memory element may store, and the processing module, module,processing circuit, and/or processing unit executes, hard coded and/oroperational instructions corresponding to at least some of the stepsand/or functions illustrated in one or more of the Figures and/ordescribed herein. Such a memory device or memory element can be includedin an article of manufacture. While the processing module, module,processing circuit, and/or processing unit device may be a generalpurpose computing device, the execution of the hard coded and/oroperational instructions by the processing module, module, processingcircuit, and/or processing unit configures such a general purposecomputing device as a special purpose computing device to implement thecorresponding steps and/or functions illustrated in one or more of theFigures and/or described herein. In particular, the hard coded and/oroperational instructions by the processing module, module, processingcircuit, and/or processing unit implement acts and algorithms performedby the processing module, module, processing circuit, and/or processingunit. Such acts and algorithms can be identified by name, can beillustrated via flowchart and/or described in words.

One or more embodiments have been described above with the aid of methodsteps illustrating the performance of specified functions andrelationships thereof. The boundaries and sequence of these functionalbuilding blocks and method steps have been arbitrarily defined hereinfor convenience of description. Alternate boundaries and sequences canbe defined so long as the specified functions and relationships areappropriately performed. Any such alternate boundaries or sequences arethus within the scope and spirit of the claims. Further, the boundariesof these functional building blocks have been arbitrarily defined forconvenience of description. Alternate boundaries could be defined aslong as the certain significant functions are appropriately performed.Similarly, flow diagram blocks may also have been arbitrarily definedherein to illustrate certain significant functionality.

To the extent used, the flow diagram block boundaries and sequence couldhave been defined otherwise and still perform the certain significantfunctionality. Such alternate definitions of both functional buildingblocks and flow diagram blocks and sequences are thus within the scopeand spirit of the claims. One of average skill in the art will alsorecognize that the functional building blocks, and other illustrativeblocks, modules and components herein, can be implemented as illustratedor by discrete components, application specific integrated circuits,processors executing appropriate software and the like or anycombination thereof.

In addition, a flow diagram may include a “start” and/or “continue”indication. The “start” and “continue” indications reflect that thesteps presented can optionally be incorporated in or otherwise used inconjunction with other routines. In this context, “start” indicates thebeginning of the first step presented and may be preceded by otheractivities not specifically shown. Further, the “continue” indicationreflects that the steps presented may be performed multiple times and/ormay be succeeded by other activities not specifically shown. Further,while a flow diagram indicates a particular ordering of steps, otherorderings are likewise possible provided that the principles ofcausality are maintained.

The one or more embodiments are used herein to illustrate one or moreaspects, one or more features, one or more concepts, and/or one or moreexamples. A physical embodiment of an apparatus, an article ofmanufacture, a machine, and/or of a process may include one or more ofthe aspects, features, concepts, examples, etc. described with referenceto one or more of the embodiments discussed herein. Further, from figureto figure, the embodiments may incorporate the same or similarly namedfunctions, steps, modules, etc. that may use the same or differentreference numbers and, as such, the functions, steps, modules, etc. maybe the same or similar functions, steps, modules, etc. or differentones.

The term “system” is used in the description of one or more of theembodiments. A system implements one or more functions via a device suchas a processor or other processing device or other hardware that mayinclude or operate in association with a memory that stores operationalinstructions. A system may operate independently and/or in conjunctionwith software and/or firmware. As also used herein, a system may containone or more sub-system, each of which may be one or more systems.

As may further be used herein, a computer readable memory includes oneor more memory elements. A memory element may be a separate memorydevice, multiple memory devices, or a set of memory locations within amemory device. Such a memory device may be a read-only memory, randomaccess memory, volatile memory, non-volatile memory, static memory,dynamic memory, flash memory, cache memory, and/or any device thatstores digital information. The memory device may be in a form a solidstate memory, a hard drive memory, cloud memory, thumb drive, servermemory, computing device memory, and/or other physical medium forstoring digital information.

While particular combinations of various functions and features of theone or more embodiments have been expressly described herein, othercombinations of these features and functions are likewise possible. Thepresent disclosure is not limited by the particular examples disclosedherein and expressly incorporates these other combinations.

What is claimed is:
 1. An electrocardiogram (ECG) interpretation system,comprising: at least one processor; and a memory that stores operationalinstructions that, when executed by the at least one processor, causethe ECG interpretation system to: receive, via a receiver, a capturedimage of an ECG printout; perform a waveform detection function on thecaptured image to determine a plurality of locations of a plurality ofECG waveforms in the captured image; generate a plurality of waveformimages by partitioning the captured image based on the plurality oflocations, wherein each of the plurality of waveform images includes oneof the plurality of ECG waveforms; generate a plurality of pseudo-rawECG signal data by performing a signal reconstruction function on eachof the plurality of waveform images, wherein each of the plurality ofpseudo-raw ECG signal data corresponds to one of the plurality ofwaveform images; generate diagnosis data by performing a diagnosingfunction on the plurality of pseudo-raw ECG signal data; and transmit,via a transmitter, the diagnosis data to a client device for display viaa display device.
 2. The ECG interpretation system of claim 1, whereinthe ECG printout was generated by an ECG machine based on electricalactivity recorded by the ECG machine, and wherein the captured image wasgenerated by a utilizing an image capture device to generate thecaptured image from the ECG printout.
 3. The ECG interpretation systemof claim 2, wherein the client device corresponds to a mobile device,wherein the image capture device corresponds to a camera, wherein thecamera is coupled to the mobile device, wherein the captured image istransmitted to the ECG interpretation system by the mobile device. 4.The ECG interpretation system of claim 2, wherein the image capturedevice corresponds to a scanner.
 5. The ECG interpretation system ofclaim 1, wherein the waveform detection function utilizes a computervision model trained on a training set of waveform images.
 6. The ECGinterpretation system of claim 1, wherein the waveform detectionfunction utilizes standardized layout information, wherein thestandardized layout information indicates relative positions ofwaveforms in standard ECG printouts.
 7. The ECG interpretation system ofclaim 6, wherein the standardized layout information indicates relativelocations of exactly twelve waveforms, wherein each of the exactlytwelve waveforms corresponds to one of twelve ECG leads, and wherein theplurality of waveform images includes exactly twelve waveform imagescorresponding to the twelve ECG leads.
 8. The ECG interpretation systemof claim 1, wherein the waveform detection function utilizesstandardized formatting feature information, wherein the standardizedformatting feature information indicates at least one relative positionof at least one additional feature in standard ECG printouts, andwherein the performing the waveform detection function includesdetecting the at least one additional feature in the ECG printout. 9.The ECG interpretation system of claim 1, wherein the signalreconstruction function determines a plurality of discrete points forone of the plurality of waveform images, and wherein a corresponding oneof the plurality of pseudo-raw ECG signal data includes the plurality ofdiscrete points, wherein the diagnosing function is trained on atraining set of raw ECG signal data, and wherein each raw ECG signaldata in the training set includes discrete points characterizing the rawECG signal data.
 10. The ECG interpretation system of claim 1, whereinperforming the diagnosing function on the plurality of pseudo-raw ECGsignal data includes: generating twelve sub-diagnosis data, wherein eachof the twelve sub-diagnosis data is generated by performing one oftwelve inference functions on a corresponding one of twelve pseudo-rawECG signal data, wherein each of the twelve inference functionscorresponds to one of twelve ECG leads and is trained on a training setof raw ECG signal data for the corresponding one of the twelve ECGleads; and performing a final inference function on the twelvesub-diagnosis data to generate the diagnosis data.
 11. The ECGinterpretation system of claim 1, wherein the captured image is receivedfrom the client device, wherein the ECG printout corresponds to apatient, and wherein the client device is associated with the patient.12. The ECG interpretation system of claim 1, wherein the captured imageis received from the client device, wherein the ECG printout correspondsto a patient, and wherein the client device is associated with a medicalprofessional responsible for the patient.
 13. The ECG interpretationsystem of claim 1, wherein the ECG printout is received in an firstemail transmission from a first email address associated with arequestor to a second email address associated with the ECGinterpretation system, wherein the diagnosis data is transmitted in asecond email transmission from the second email address to the firstemail address, and wherein the requestor is a user of the client device.14. The ECG interpretation system of claim 13, wherein the second emailtransmission includes a document for display to the requestor via theclient device, and wherein the document includes the diagnosis data andfurther includes the plurality of pseudo-raw ECG signal data.
 15. TheECG interpretation system of claim 1, wherein the diagnosis data isdisplayed via the display device in conjunction with the captured image.16. The ECG interpretation system of claim 1, wherein the diagnosis datais displayed via the display device in conjunction with the plurality ofpseudo-raw ECG signal data via a user interface.
 17. The ECGinterpretation system of claim 16, wherein the diagnosis data indicatesat least one abnormality in at least one of the plurality of pseudo-rawECG signal data, and wherein the at least one abnormality is visuallyindicated in the display of the at least one of the plurality ofpseudo-raw ECG signal data.
 18. The ECG interpretation system of claim17, wherein the operational instructions, when executed by the at leastone processor, further cause the ECG interpretation system to:generating normal pseudo-raw ECG signal data by removing the at leastone abnormality from the at least one of the plurality of pseudo-raw ECGsignal data, wherein the normal pseudo-raw ECG signal data is displayedvia the display device in conjunction with the diagnosis data and theplurality of pseudo-raw ECG signal data.
 19. The ECG interpretationsystem of claim 1, wherein the operational instructions, when executedby the at least one processor, further cause the ECG interpretationsystem to: receive, via the receiver, a second captured image of apatient's skin; generate second diagnosis data by performing a seconddiagnosing function on the second captured image, wherein the seconddiagnosing function utilizes a computer vision model trained on aplurality of images of skin corresponding to a plurality of skinconditions; and transmit, via the transmitter, the second diagnosis datato a second client device for display via a second display device.
 20. Amethod for execution by an ECG interpretation system, the methodcomprising: receiving, via a receiver, a captured image of an ECGprintout; performing a waveform detection function on the captured imageto determine a plurality of locations of a plurality of ECG waveforms inthe captured image; generating a plurality of waveform images bypartitioning the captured image based on the plurality of locations,wherein each of the plurality of waveform images includes one of theplurality of ECG waveforms; generating a plurality of pseudo-raw ECGsignal data by performing a signal reconstruction function on each ofthe plurality of waveform images, wherein each of the plurality ofpseudo-raw ECG signal data corresponds to one of the plurality ofwaveform images; generating diagnosis data by performing a diagnosingfunction on the plurality of pseudo-raw ECG signal data; andtransmitting, via a transmitter, the diagnosis data to a client devicefor display via a display device.